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SUMMARY 


First  generation  demonstration  programs  such  as  A-10  and 
B-l  provided  useful  information  and  experience  in  the  deter- 
mination of  the  NDE  capability  of  a facility.  A large  amount  of 
reliability  data  has  been  generated  since  these  demonstration 
programs  which  should  aid  in  the  development  of  a more  valid 
second  generation  demonstration  program.  The  purpose  of  the 
present  program  is  to  develop  a model  to  translate  the  non- 
destructive evaluation  (NDE)  capabilities  assessed  from  alumi- 
num and  steel  specimens  with  simple  geometries  to  equivalent 
detection  sensitivity  for  specimens  with  complex  geometries. 

Five  NDE  methods  are  included  in  this  study:  ultrasonics,  eddy 
current,  magnetic  particle,  penetrant  and  radiography. 

The  NDE  reliability  data  base  compiled  under  a previous 
NASA  program  was  enlarged,  updated  and  edited.  Adaptive 
learning  techniques  and  a linear  regression  analysis  method 
were  used  to  establish  parametric  relationships  between  in- 
spection sensitivity  and  NDE  parameters.  Based  on  results 
from  the  parametric  study,  translation  models  were  developed  to 
translate  inspection  results  obtained  from  flat  plate  specimens 
to  equivalent  results  on  fillet  areas,  weldments,  and  tandem 
T specimens.  The  parametric  relationships  and  translation 
models  developed  in  this  program  were  limited  to  the  material 
of  aluminum  and  the  NDE  techniques  of  ultrasonics,  eddy  currents 
and  penetrant.  The  limitations  were  due  to  the  scope  of  the 
existing  data.  Overall,  the  translation  model  from  aluminum 
flat  plate  to  fillet  specimens  was  most  successful.  Data 
deficiencies  were  identified  during  the  model  development.  An 
optimum  demonstration  program  was  designed  to  be  used  as  a 
guideline  for  future  validation  and  NDE  facility  qualification 
programs . 
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SECTION  I 
INTRODUCTION 


The  design  damage  tolerance  requirements  (MIL-A-83444) 
specify  the  maximum  allowable  initial  flaw  sizes  in  aircraft 
structure.  The  damage  tolerance  design  approach  is  based  on 
a sound  fracture  mechanics  principles.  It  assumes  that  an  air- 
craft hardware  item  contains  an  initial  defect  or  flaw  of 
some  minimum  size  which  can  be  reliably  detected  with  a 
demonstrated  NDE  capability.  The  Aircraft  Structural  Integrity 
Program  (MIL-STD- 1530)  specifies  that  a NDE  demonstration 
program  shall  be  performed  to  verify  that  all  flaws  equal  to 
or  greater  than  the  design  flaw  size  will  be  detected  to  the 
required  reliability  and  confidence  level.  The  demonstration 
program  is  required  if,  and  only  if,  the  designer  elects  not 
to  accept  the  design  flaw  size  specified  in  MIL-A-83444. 

Demonstration  programs  conducted  in  the  past  produced 
useful  information  as  required  by  their  respective  specific 
objective.  These  first  generation  programs  may  not  have  been 
totally  adequate  from  the  standpoints  of  flaw  size  distribution, 
NDE  parameters  identification,  and  sensitivity  interpretation. 

The  A-10  and  B-l  demonstration  programs  were  among  these  first 
generation  programs  that  provided  useful  knowledge  and  experiences 
in  the  attempt  to  determine  the  NDE  capability  of  a facility. 

A large  amount  of  data  in  the  area  of  NDE  reliability  has  been 
generated  after  these  demonstration  programs  were  completed. 

The  knowledge  gained  from  these  studies  allow  us  to 
develop  a more  valid  second  generation  demonstration  program. 

The  need  for  the  design  of  a more  valid  demonstration 
program  is  illustrated  by  an  example  of  the  B-l  demonstration 
program.  In  that  program  flat  plate  specimens  were  used  which 
contained  fatigue  cracks  grown  under  cyclic  fatigue  loads. 
Specimens  with  this  simple  geometry  have  the  advantages  of  low 
cost  as  well  as  ease  in  crack  growth  and  flaw  size  control. 
However,  in  actual  B-l  structures  there  are  fracture  critical 
areas  in  fillets  and  curved  surfaces  for  which  no  reliability 
data  was  available  at  that  time.  The  NDE  capability  demonstrated 
by  a facility  on  flat  plate  specimens  should  be  translated  to 
the  equivalent  capability  on  specimens  with  complex  geometries. 
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Such  a translation  was  made  possible  by  two  programs  sponsored 
by  NASA  Johnson  Space  Center(l»2).  In  these  programs,  relia- 
bility data  were  generated  by  the  same  facility  on  specimens 
with  flat  plate  and  complex  geometries  to  allow  a valid  comparison 
of  the  inspection  sensitivity. 

The  data  base  for  the  NDE  parametric  study  and  translation 
model  development  consists  of  twenty  sets  of  relevant  NDE 
reliability  data  compiled  from  previous  research  programs  con- 
ducted in  different  aerospace  companies.  A computerized  relia- 
bility data  bank  was  compiled  by  General  Dynamics  and  Vanderbilt 
University  under  a program  sponsored  by  the  NASA  Lewis  Research 
Center(3).  in  this  program  twenty  sets  of  relevant  NDE  relia- 
bility data  were  identified,  collected,  compiled,  and  categorized. 
Three  on-going  programs  generating  relevant  data  were  also 
identified.  The  criteria  for  data  selection  for  statistical 
analysis  consideration  were  formulated  on  the  basis  of  the 
completeness  of  pertinent  NDE  parameter  records,  requirement 
for  instrument  calibration,  verification  of  flaw  dimensions, 
and  other  relevant  factors.  A comprehensive  computer  program 
was  prepared  to  calculate  the  probability  of  flaw  detection  (POD)  at 
50  and  95  percent  confidence  levels  by  using  the  binomial 
statistical  method. 

Three  data  cumulation  procedures  were  used  in  plotting  the 
POD  as  a function  of  flaw  size:  (1)  range  interval  (RI) , 

(2)  overlapping  60  points  (OSP) , and  (3)  optimized  probability 
method  (0PM) . In  the  RI  method  the  data  were  separated  into 
groups  of  equal  flaw  size  increments.  The  probability  of  de- 
tection at  the  onesided  lower  confidence  limit  was  computed  for 
each  group  separately  and  plotted  as  a histogram  bar.  The  OSP 
method  combined  detection  results  for  the  largest  60  cracks  and 
plotted  the  POD  for  this  interval  at  the  largest  flaw  size. 

The  next  data  increment  was  obtained  by  starting  at  the  median 
flaw  size  of  the  first  interval  and  combining  the  data  for  the 
next  smaller  60  cracks.  The  POD  of  the  second  set  was  plotted 
at  its  largest  flaw  size  and  the  process  was  repeated  until  all 
data  were  combined.  In  the  0PM  method  the  ordered  NDI  data 
were  grouped  into  a given  number  of  intervals  of  successively 
increasing  size  range.  POD  for  the  largest  size  range  and  its 
successive  combinations  with  the  smaller  sizes  were  calculated. 

The  largest  value  of  the  POD  was  plotted  at  the  largest  flaw 
size  contained  in  the  corresponding  composite  grouping.  The 
largest  flaw  size  interval  was  then  removed  from  consideration 
and  the  procedure  was  repeated  starting  from  the  next  to  largest 
flaw  size  grouping.  The  latter  was  repeated  until  the  given 
number  of  intervals  of  POD  was  plotted. 
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Seven  sets  of  reliability  data  collected  in  the  NASA 
program  were  statistically  analyzed  by  using  the  cumulative 
schemes  described  in  the  above  paragraph.  These  seven  sets 
of  data  met  the  selection  criteria  established  in  that  program. 
Among  these  seven  sets  were  data  collected  on  flat  plate 
specimens  containing  fatigue  cracks  in  the  NDI  studies  initiated 
by  NASA  Johnson  Space  Center. 

With  the  availability  of  more  and  more  NDE  reliability  data, 
it  has  become  necessary  to  start  examining  the  parameters  that 
can  potentially  influence  the  inspection  results.  These  parameters 
need  to  be  identified  and  their  relative  influence  on  the  ins- 
pection results  determined  by  systematic  parametric  studies. 
Specimen  geometry,  defect  type  and  orientation,  surface  condition, 
inspection  environment,  human  factors  and  the  statistical  methods 
chosen  for  the  capability  determination  are  but  a few  of  the  NDE 
parameters  that  may  influence  the  inspection  results.  A model 
to  translate  results  obtained  from  practical  test  specimens  to 
complex  aircraft  hardware  needs  to  be  developed.  Such  a model 
will  be  in  the  form  of  a family  of  probability  of  detection  (POD) 
curves  plotted  as  a function  of  flaw  size  with  the  specimen 
geometry  as  the  independent  parameter.  This  family  of  curves 
will  provide  a comparison  of  POD  for  any  two  specimen  geometries 
at  a specific  flaw  size.  Equivalently,  translation  tables  may 
be  constructed  in  which  the  differences  in  POD  for  any  two 
specimen  geometries  are  provided. 
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SECTION  II 


PROGRAM  OBJECTIVE 
AND  APPROACH 


The  overall  objective  of  this  program  was  to  develop 
a moot  to  translate  the  NDE  capabilities  assessed  from 
alumi  and  steel  specimens  with  simple  geometries  to 
equiVvti  i-t  detection  sensitivity  for  specimens  with  complex 
geometi  es.  The  specific  objectives  were: 

o Enlarge  the  NDE  reliability  data  base  compiled 
under  a previous  NASA  program 

o Develop  analytic  techniques  to  analyze  the 
reliability  data 

o Establish  the  parametric  relationships  between 
inspection  sensitivity  and  NDE  parameters 

o Identify  data  deficiencies  in  the  reliability 
data  base  and  outline  experiments  to  overcome 
the  deficiencies 

o Develop  translation  models  for  NDE  techniques 
of  ultrasonic,  penetrant,  eddy  current,  magnetic 
particle  and  X-ray 

o Design  an  optimum  demonstration  program  to 
evaluate  the  NDE  capability  of  industrial 
facilities . 

The  basic  approach  to  achieve  the  objectives  of  this 
program  was  to  use  existing  reliability  data,  identify  those 
data  containing  inspection  reliability  results  on  specimens 
with  simple  and  complex  shapes  obtained  under  similar  con- 
ditions, and  to  build  translation  models  based  on  the  available 
data.  However,  the  existing  reliability  data  did  not  have  a 
systematic  variation  in  NDE  parameters  to  allow  the  variables 
to  be  readily  separated  in  parametric  studies.  Therefore, 
statistical  methods  must  be  developed  or  adapted  to  analyze 
the  data  for  the  parametric  study  and  translation  model 
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development.  A standard  linear  regression  method  and  an 
adaptive  learning  technique  were  used  in  this  program  in 
the  statistical  analyses.  The  analyses  were  conducted  by 
computer  codes  applicable  to  an  IBM  370  and  a PDP  11/45 
digital  computer.  The  NDE  parameters  in  a given  data  set  or 
subset  were  treated  as  independent  variables  while  the 
probabilities  of  detection  associated  with  the  crack  size 
ranges  in  that  data  set  or  subset  were  considered  the 
dependent  outputs  in  the  parametric  study.  When  the  specimen 
geometry  was  considered  as  the  independent  variable  while 
other  NDE  parameters  were  held  constant,  the  variation  in 
the  POD  for  the  specimen  geometries  formed  the  basis  of  a 
translation  model.  Knowing  how  the  NDE  parameters  affected 
the  inspection  sensitivity  and  how  much  difference  in 
inspection  sensitivity  existed  between  specimens  with  simple 
geometries  and  specimens  with  complex  configurations,  an 
optimum  demonstration  program  could  be  designed.  This  optimum 
demonstration  program  defined  the  important  NDE  parameters 
to  be  maintained  in  the  execution  of  the  program,  and 
interpreted  results  obtained  in  the  program  for  simple 
test  specimens  in  terms  of  equivalent  NDE  capability  as 
applied  to  more  complex  components. 

The  results  obtained  in  this  program  were  limited 
mostly  to  aluminum  and  the  NDE  techniques  of  ultrasonic, 
penetrant  and  eddy  current.  The  limitation  was  due  to  the 
inadequacy  of  the  existing  data  for  steel  and  for  the  two 
NDE  techniques  of  X-ray  and  magnetic  particle.  Titanium  was 
not  selected  as  one  of  the  materials  to  be  analyzed  in  this 
program  on  account  of  data  deficiency  for  this  material. 
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SECTION  III 


DATA  BASE 


The  NDT  reliability  data  base  developed  by  General  Dynamics 
under  a NASA  contract (1)  has  been  enlarged,  updated  and  modified 
to  serve  as  a basis  for  the  parametric  study  and  translation 
model  development  in  this  program.  Seventy  additional  data  sets 
were  entered  into  the  computer  data  bank,  bringing  the  total 
number  of  data  sets  to  222.  The  additional  data  were  generated 
by  Martin  Marietta  Corporation/Aerospace  Division^).  These 
data  consisted  of  NDT  inspection  results  on  aluminum  specimens 
in  the  configuration  of  integrally  stiffened  panels  and  welded 
panels.  Fatigue  cracks  in  the  fillet  area  of  the  integrally 
stiffened  panels  and  weld  joints  as  well  as  lack  of  penetration 
(LOP)  in  the  welded  panels  constituted  the  flaw  types  in  the 
specimens.  Specimen  history  included  as  machined  and  chemically 
etched  surface,  scarfed  and  unscarfed  crowns,  and  proof  loaded 
to  90  percent  of  the  yield  strength.  NDT  methods  used  in  the 
inspection  were  liquid  penetrant,  ultrasonic,  eddy  current  and 
X-radiography  techniques.  A complete  listing  of  all  the  data 
sets  in  the  data  bank  is  tabulated  in  Table  1. 

The  additional  data  sets  entered  into  the  computer  data 
bank  supplemented  the  basic  reliability  data  needed  for  the  NDT 
parameter  relationship  and  translation  model  development. 

Similar  data  were  generated  by  the  same  company  under  an  earlier 
NASA  sponsored  research  program  using  flat  plates  as  specimens ( 3) . 
The  facilities  and  procedures  of  the  inspections  were  essentially 
the  same  for  the  two  programs.  Two  other  companies,  Rockwell 
International/Space  Division  and  General  Dynamic s/Convair  Division 
also  participated  in  the  earlier  program.  The  34  data  sets 
compiled  by  the  three  participating  companies  in  the  earlier 
program  plus  the  70  data  sets  compiled  by  Martin  Marietta  in  the 
latter  program  are  well  documented  and  well  balanced  statistically 
The  categories  and  individual  parameters  for  each  data  set  are 
shown  in  the  data  set  listing  in  Table  1. 

Several  data  sets  analyzed  in  the  NASA  progran/^  were  not 
used  in  the  translation  model  development.  Some  of  the  data 
collected  in  the  NASA  program  lacked  information  on  NDE  parameters. 
Some  data  were  well  documented  but  the  POD  for  the  flawed  speci- 
mens were  so  high  that  a curve  cannot  be  plotted  in  the  process 
of  building  a translation  model. 
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Considerable  effort  has  been  expended  in  the  modification 
and  updating  of  the  data  file  format  and  contents  for  their 
utilization  in  this  program.  Continual  corrections  have  been 
made  from  surveys  and  personal  contacts  to  improve  the  accuracy 
of  the  data  and  associated  NDT  parameters.  The  cover  sheet 
format  identifying  the  pertinent  information  on  each  data  set 
has  been  modified  to  expedite  data  pooling  and  information 
retrieval.  Modifications  to  the  entire  data  bank  system  were 
also  made  to  provide  flexibility  for  future  additions.  An 
example  of  the  cover  sheets  for  the  ultrasonic  method  is  given 
in  Table  2.  The  data  set  number  (DSN)  and  inspection  date 
comprise  the  heading  in  each  table.  The  last  entry  in  the 
heading  shows  the  date  on  which  the  cover  sheets  for  the  NDT 
techniques  of  eddy  current,  radiography,  magnetic  particle  and 
penetrant  methods  are  presented  in  Tables  A-l  to  A-4  respec- 
tively in  Appendix  A. 


An  industry  NDE  facility  survey  was  conducted  during  the 
program  period.  The  objectives  of  the  survey  were:  (1)  to 
acquire  detailed  information  on  the  data  contributed  by  each 
facility,  (2)  to  obtain  general  information  on  NDT  equipment, 
reference  standards,  and  inspector  levels  in  each  facility,  and 
(3)  to  solicit  opinions  from  NDT  personnel  in  each  facility  in 
the  design  of  an  optimum  demonstration  program.  Martin  Marietta 
Corporation/Aerospace  Division,  the  B-l  and  the  Space  Division 
of  Rockwell  International  Corporation  were  objects  of  the  survey. 
Useful  information  pertaining  to  the  B-l  demonstration  program 
such  as  the  history,  specimen  type,  flaw  type  and  inspection 
procedures  were  furnished  by  NDE  personnel  at  the  B-l  Division 
of  the  Rockwell  International  Corporation.  Details  of  the 
penetrant  inspection  in  that  demonstration  program  were  discussed 
during  the  survey.  One  of  the  important  conclusions  from  the 
discussion  was  that  the  inspection  capability  should  not  be 
treated  as  a monotonic  function  of  the  sensitivity  of  the 
penetrant.  Whether  the  fatigue  crack  was  the  proper  flaw  type 
to  be  used  in  specimens  for  demonstration  program  designed  to 
verify  production  inspection  was  also  discussed.  At  the  Space 
Division  of  the  Rockwell  International  Corporation,  the  Space 
Shuttle  NDE  demonstration  program  was  discussed  in  detail. 
Procedures  for  certifying  penetrants  for  the  Space  Shuttle 
inspection  were  provided  by  the  Space  Division  personnel. 

Details  on  the  fabrication  of  the  flat  plate  specimens  and 
specimens  with  complex  geometries  were  provided  by  personnel  at 
the  Aerospace  Division  of  Martin  Marietta  Corporation. 
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TABLE  2 RELEVANT  NDE  PARAMETERS  FOR  ULTRASONIC  METHOD 
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From  the  facilities  included  in  the  survey,  first  hand 
information  was  obtained  on  the  NDE  equipment,  reference 
standards,  and  inspector  levels.  It  was  found  that  most  of 
the  aerospace  companies  were  purchasing  newer  automated  NDE 
equipment  to  replace  the  older  manual  models.  The  choice  of 
reference  standards  was  still  a baffling  problem  to  most  of 
the  NDE  community.  The  results  of  the  survey  on  inspector  level 
strengthened  the  previous  opinion  that  nondestructive  testing 
inspectors  at  the  same  level  of  technical  classification  may 
differ  significantly  in  the  degree  of  skill.  Before  conducting 
a demonstration,  the  inspectors  were  usually  screened  or  updated 
in  training.  The  information  acquired  in  the  survey  of  the 
facilities,  as  well  as  similar  information  gathered  in-house 
at  the  Fort  Worth  Division  of  General  Dynamics  Corporation, 
formed  the  background  data  for  the  translation  model  development 
and  served  to  mold  the  design  of  the  optimum  demonstration 
program.  Valuable  opinions  were  obtained  from  personnel  in 
these  facilities  in  the  areas  of  flaw  type,  flaw  location,  NDE 
methods  applicable  to  critical  flaws,  and  the  specimen  geometry 
for  most  commonly  occurring  flaws  in  certain  types  of  aircraft 
(e.g.  B-l).  Corrections  in  the  NDE  reliability  data  were  made  in 
some  instances  which  had  the  impact  of  increasing  the  accuracy 
of  the  NDE  parametric  relationships  and  translation  model  develop- 
ment in  this  program. 


Several  potential  sources  for  the  expansion  of  the  relia- 
bility data  bank  have  been  identified  during  the  course  of  the 
program.  The  NDE  facilities  participating  in  the  NASA  Space 
Shuttle  NDE  Program  generated  additional  data  on  steel  components. 
Much  needed  magnetic  particle  inspection  data  were  included  in 
this  category  of  data.  A program  titled  "Reliability  of  Non- 
destructive Inspection  on  Aircraft  Structures"  is  currently 
being  conducted  at  Lockheed-Georgia  Company  under  the  sponsorship 
of  AFLC  at  Kelly  Air  Force  Base.  The  field  inspection  results 
from  the  NDI  reliability  program  will  be  available  in  1978.  These 
inspection  results  will  supplement  the  data  bank  file  in  the  field 
and  depot  inspection  category.  Results  from  an  on-going  AF  program 
"Quantitative  Evaluation  of  Penetrant  Inspection  Materials  and 
Procedures"  (Contract  No.  F33615-76-C-5166)  will  also  be  available 
in  1977.  In  addition  to  the  above  on-going  programs,  the  analysis 
of  results  from  an  ASNT-sponsored  round  robin  program  on  ultra- 
sonic inspection  of  steel  and  aluminum  parts  will  be  completed  in 
the  fall  of  1977.  The  reliability  data  from  these  four  programs 
will  substantially  enlarge  the  existing  data  file. 
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SECTION  IV 


METHODOLOGY 


In  addition  to  a straightforward  method  of  point  estimate 
plot  of  the  probability  of  detection,  two  data  fitting  tech- 
niques were  used  in  the  analysis.  These  two  techniques,  linear 
regression  analysis  and  adaptive  learning,  provided  the  mathe- 
matical tools  in  fitting  the  reliability  data  in  well  behaved 
probability  of  detection  curves.  The  parametric  study  and 
translation  model  development  were  based  on  the  successful 
applications  of  these  techniques.  In  the  following  sections, 
a brief  introduction  of  each  of  the  two  techniques  will  be 
given.  Details  of  the  methodologies  and  associated  computer 
software  programs  are  presented  in  Appendix  B. 


4.1  LINEAR  REGRESSION  ANALYSIS 

The  development  of  the  multiple  linear  regression  analysis 
technique  and  the  development  procedure  of  a linear  translation 
model  will  be  discussed  in  the  following  paragraphs. 

4.1.1  Introduction  to  Multiple  Linear  Regression 

The  purpose  of  using  multiple  linear  regression  techniques 
is  to  assist  in  the  development  of  an  informative  model  for  the 
POD  as  a function  of  several  NDE  variables.  A good  model  must 
"fit"  the  data  reasonably  well.  (However,  undue  concern  about 
"fitness"  should  be  avoided  because  it  is  easy  to  construct 
models  which  fit  sets  of  data  with  no  error  and  yet  are  almost 
totally  uninformative) . It  should  isolate  the  effects  of  the 
various  NDE  variables  as  much  as  possible  and  for  this  specific 
application  its  form  must  be  suitable  for  transfer  function 
development . 

The  general  principle  that  motivates  regression  analysis  is 
that  the  variability  in  the  observations  of  the  dependent  variable 
(POD)  can  be  narrowed  (or  accounted  for  when  the  variability  is 
measured  numerically)  by  knowledge  of  one  or  more  of  the  indepen- 
dent variables  (crack  length,  specimen  geometry,  specimen  thick- 
ness, etc.)  that  were  observed  along  with  the  POD.  It  should  be 
kept  in  mind  that  the  NDE  personnel  should  be  in  control  of 
building  the  model  and  his  knowledge  of  NDE  variables  should  be 
reflected  in  the  final  model  because  regression  procedures  do  not 
automatically  produce  meaningful  and  valid  results. 


17 


4.1.2  Formal  Development 

The  general  linear  model  can  be  presented  most  concisely 
in  terms  of  matrix  operations  as  follows: 

Y = X f)  + e 

where  Y is  a N x 1 vector  of  observations  of  the  dependent  variable, 

X is  a N x P matrix  which  specifies  the  form  in  which  P 
independent  variables  are  related  to  Y, 

(3  is  a P x 1 vector  of  unknown  coefficients  which  are  to  be 
estimated  from  the  data,  and 

e is  a N x 1 vector  of  errors  which  can  only  be  observed 
explicitly  if  [i  is  known. 

There  are  several  assumptions  underlying  a regression 
analysis,  as  follow: 

1.  The  components  of  X are  constants  which  are  measured 
without  error. 

2.  E(e)  = 0,  that  is  the  average  error  is  0. 

3.  COV(e)  = where  <x 2 is  a constant  and  1^  is  the 

N x N identity  matrix  (the  covariance  matrix  has  the 
variances  of  the  errors  on  the  main  diagonal  and  the 
covairances  of  the  errors  in  the  off-diagonal  positions, 
so  the  above  form  of  the  covariance  matrix  is  a method 
of  stating  the  error  variances  are  homogeneous  and 

the  errors  are  uncorrelated). 

The  additional  assumption  of  normally  distributed  errors 
permits  one  to  test  hypotheses  about  the  coefficients  (components 
of  (3  ) and  to  establish  confidence  limits  or  prediction  intervals 
about  the  regression  line  or  surface. 

The  criterion  which  is  used  to  determine  the  coefficients 
is  the  least  squares  criterion.  Some  notation  and  basic  facts 
are  needed  to  outline  the  least  squares  criterion: 

A 

Y = the  value  predicted  by  the  model  for  Y. 

A 

Y - Y = the  amount  the  model  misses  in  predicting  Y,  called 
a residual,  R, 


R = Y - 

A 

P = the  estimate  of  p , 


R'R  = the  sum  of  the  squares  of  the  components  of  R where 
a primed  superscript  of  a matrix  indicates  the  transpose  of  the 
matrix. 

The  least  squares  criterion  is  to  select  the  components  of 
so  that  R'R  is  a minimum.  After  algebraic  manipulation  it  can 
be  shown  that : 

P = (X'X)  1 X'Y  whenever  X'X  has  an  inverse^^. 

R 1 R 

In  some  instances  it  is  appropriate  to  use  as  an  esti- 
mator of  0-2  (this  will  be  discussed  in  more  detail  in  a 

later  section) . 

Since  probabilities  are  being  predicted  a truncation  rule 
must  be  used  which  truncates  predicted  probabilities  larger  than 
1,  to  1 exactly  and  which  truncates  negative  probabilities  to  0. 

A simple  example  will  illustrate  some  of  these  matrix 
operations.  Suppose  there  are  5 observations  of  (X,Y)  and  the 
linear  relationship  between  X and  Y is  to  be  determined.  The  model 
is 


Y1 

1 Xl~ 

ei 

Y2 

1 x2 

e2 

Y3 

= 

1 x3 

PV)  + 

e3 

Y4 

1 X4 

e4 

Y5 

} X5_ 

_e5_ 

The  transpose  of  the  matrix  X is : X ' = i x X X3  X 
The  product  of  the  matrices  X'  and  X is: 


5 IX. 
1 


(X’X) 


2 

IX.  IX. 

1 1 
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The  inverse  of  the  product  is: 


4.1.3  Preliminaries  to  Model  Specification 

Some  terminology  and  basic  concepts  from  experimental  statis- 
tics, illustrated  by  an  example  from  the  current  data  base,  are 
needed  before  the  problem  of  model  specification  is  attacked  in 
the  next  section.  Suppose  that  the  only  variables  which  have  an 
effect  on  the  POD  are  crack  length  (9  intervals)  and  specimen 
geometry  (geometries  1,  2 and  3).  There  are  27  combinations  of 
crack  length  and  geometry  which  can  be  displayed  in  a two-way 
table  (a  specific  combination  in  the  table  will  be  referred  to  as 
a cell) . The  POD  observed  in  each  cell  is  assumed  to  be  the  sum 
of  several  components.  A constant  which  is  the  same  in  all  cells 
is  the  first  component.  The  next  components  are  a contribution 
from  the  crack  length  interval  the  cell  is  in,  a contribution 
from  the  geometry  the  cell  is  in,  a contribution  from  the  com- 
bination of  geometry  and  crack  length  the  cell  is  in  and  finally 
an  error  which  cannot  be  observed.  To  illustrate  this  point 
analytically,  let  H be  the  constant,  ct±,  i = 1,2,..., 9 be  the 
crack  length  contribution,  £j,j  = 1,2,3  be  the  geometry  contri- 
bution, ( a p )^j  and  eij > i = 1,2,..., 9,  j = 1,2,3  be  the  crack 
length  geometry  combination  contribution  and  error  contribution 
respectively.  Table  3 summarizes  the  above  formulation. 

The  a^»  s and  i s are  referred  to  as  main  variable  effects 
because  they  are  usecT  for  comparing  the  PODs  of  one  crack  length 
with  the  PODs  of  another  crack  length,  or  for  comparing  PODs  on 
specimens  with  different  geometries.  The  (ay?  )ij  ' s are  referred 
to  as  two-factor  interactions.  If  all  the  variables  are  fixed, 
except  one,  and  if  comparisons  are  made  among  the  levels  of  that 
variable  then  these  comparisons  can  be  called  simple  comparisons. 
For  example,  the  difference  in  PODs  for  crack  length  3 and  crack 
length  4 at  geometry  1 is  a simple  comparison.  There  are  a very 
large  number  of  simple  comparisons  that  can  be  made.  The  only 
theoretical  problem  with  simple  comparisons  is  that  the  inferences 
from  them  are  very  restricted  or  narrow.  Comparisons  of  levels 
of  one  variable  "averaged"  over  all  the  levels  of  all  the  other 
variables,  can  be  called  main  comparisons.  In  the  above  example 
if  the  POD  for  crack  length  3 is  compared  with  the  POD  for  crack 
length  4,  averaged  over  all  three  geometries  then  the  comparison 
is  a main  comparison  with  much  broader  inference  possible. 

Main  comparisons  must  be  used  with  caution  when  interactions 
are  present,  for  example,  the  MEAN  of  the  PODs  at  crack  length  3 
(neglecting  error  terms)  is  y + a-  + ^ 1 + ^ 2 + P 3 + 

J 3 

( a P ) 21  + (a  & ^32  + ( u P ) 32>  while  the  MEAN  of  the  PODs  at 
3 
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TABLE  3 

SUMMARY  OF  CONTRIBUTIONS  IN  LINEAR  REGRESSION  ANALYSIS 


Geometry 


crack  length  4 (neglecting  error  terms)  is  M + a4  + 

B i + /?2  + /*3  («  P >4i  + («  /3  >42  + ( « £ >43  # The  difference 

3 + 3 


between  these  two  MEANS  is 


a4 


+ P >31  + (a  ^32  + (a  **>33 

3 


( « ) 41  + (a/?)42  + (a^)43-  If  there  is  no  interaction, 
3 

this  comparison  is  exactly  a 3 - “4  as  desired,  but  if  inter- 
action is  present  the  comparison  of  MEANS  is  "contaminated"  by 
the  interaction  terms. 


When  the  data  is  examined,  it  is  seen  that  some  of  the 
effects  cannot  be  estimated  because  of  lack  of  data  in  some  cells. 
Consider  the  crack  length  and  geometry  data  (from  the  ultrasonic 
data  base) : 


Crack  Length  Interval 


1 

2 

3 

4 

5 

6 

7 

8 

9 

Average 

1 

.23 

.36 

.47 

.79 

.89 

.88 

.95 

.96 

.98 

.774 

2 

M 

M 

M 

.29 

.59 

.77 

.91 

.96 

.88 

^ -745 

3 

M 

M 

M 

.83 

.38 

.38 

.77 

.95 

.93 

.724 

(NOTE:  The  average  for  each  geometry  cannot  be  computed 

directly  from  the  table,  because  there  are  unequal 
numbers  of  observations  in  each  cell). 


Missing  ) 


(«  appears  only  in  cell  (1,2)  which  is  missing,  hence 

there  is  no  logical  way  to  estimate  ( a f3  • In  cell  (1,2) 
there  is  also  ft  , a\,  (3  3 but  they  appear  in  other  cells  which 

are  not  missing  and  thus  can  be  estimated.  A model  which  in- 
cludes parameters  that  are  not  estimateable  can  only  add  con- 
fusion to  the  final  results. 

This  data  also  illustrates  that  averaging  over  levels  of 
variables  without  accounting  for  the  pattern  of  missing  observa- 
tions can  lead  to  erroneous  conclusions.  The  MEAN  POD  for 
geometry  1 is  .774,  and  for  geometry  2 is  .745,  but  the  observa- 
tions for  a crack  lengths  1,  2 and  3 (where  the  POD  would  be 
expected  to  be  small)  are  missing  for  geometry  2.  If  there 
were  observations  available  for  these  missing  crack  lengths,  the 
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differences  that  were  observed  would  have  been  much  larger. 

The  model  of  the  form  discussed  in  this  section  accounts  for  the 
missing  observations  as  well  as  any  other  model.  Keeping  these 
concepts  and  problems  in  mind,  some  specific  models  are  discussed 
in  the  next  section. 

4.1.4  Building  the  Appropriate  Linear  Model 

Determining  the  form  and  complexity  of  the  model  are  impor- 
tant first  tasks  in  the  model  development.  Consider  the  form  of 
the  model  first.  If  there  is  some  knowledge  of  the  relationship 
between  the  POD  and  any  of  the  NDE  variables  this  should  be 
utilized  in  the  model.  If  it  were  known  that  POD  increased 
exponentially  with  crack  length,  a coefficient  could 
be  included  that  estimates  the  rate  of  exponential  increase.  For 
this  specific  problem  no  information  is  assumed  to  be  known  about 
the  form  of  the  relationship  between  any  NDE  variable  and  the  POD, 
hence  two  general  types  of  linear  models  were  considered.  These 
two  models  are  the  polynomial  regression  model  and  the  analysis 
of  variance  (AOV)  model  (discussed  and  illustrated  in  the 
previous  section)  implemented  through  regression  techniques.  An 
example  will  help  illustrate  the  similarities  and  differences 
between  the  two  models.  Suppose  a model  for  POD  as  a function  of 
only  specimen  geometry  is  desired,  and  further  suppose  there  are 
eight  specimen  geometries  in  the  data  base.  The  polynomial  re- 
gression model  written  without  the  matrix  form  would  be: 

y = 0O  + X + + • • • + ^7  X^  + e 

where, 

y = POD  observed, 

f}^,  i = 0,  ...  , 7 are  unknown  coefficients  to  be  estimated, 

X = 1,  2,  ...  ,8  depending  on  the  geometry  (other  codings 

of  the  X's  are  also  permissible), 

e = unobserved  random  error. 

The  analysis  of  variance  model  would  be: 

y = ^ + aL  IXi  + a2  IX2  + • • • + a8  IXg  + e 
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where 


y = POD  observed., 

u , a i = 1,  ...  , 8 are  unknown  coefficients  to  be 
estimated, 

Ix-,  i = 1,  ...  , 8 are  indicator  variables  equal  to  1 when 
the  geometry  is  and  equal  to  0 otherwise, 

e = unobserved  random  error. 

(NOTE:  For  any  specific  geometry  exactly  one  of  the  indicator 
variables  is  equal  to  1.) 

The  indicator  variables  are  usually  dropped  from  the  notation 
after  they  have  been  formally  introduced,  although  they  are  under- 
stood to  still  be  present.  The  AOV  model  discussed  informally  in 
the  previous  section  does  not  display  the  indicator  variables 
explicitly.  The  AOV  model  is  most  suitable  for  classification 
variables  (variables  where  the  numerical  value  assigned  to  the 
variable  is  arbitrary),  because  the  coefficients  can  be  used  in 
a straightforward  manner  to  compare  levels  of  variables  (i.e., 
comparing  flat  plate  specimens  with  integrally  stiffened  panel 
specimens).  It  should  be  noted  that  these  two  models  will  provide 
exactly  the  same  "fit"  of  the  data,  but  the  AOV  model  provides  an 
easier  and  clearer  method  of  developing  a translation  model. 

The  second  task  is  selecting  the  complexity  of  the  model 
(the  AOV  model  was  selected  in  the  previous  step).  The  general 
principle  used  is:  the  simplest  model  which  fits  the  data 
reasonably  well  and  which  is  most  suitable  for  translation  model 
development,  is  the  best  model.  The  best  method  of  estimating 
the  error  variance  cannot  be  used  (discussed  in  the  next  section) 
so  high  order  interactions  must  be  used  to  estimate  the  noise 
level.  There  are  several  two-factor  interactions  that  are  ex- 
pected to  be  large  (those  with  crack  length)  and  several  which 
are  important  for  translation  model  development  (those  variables 
interacting  with  specimen  geometry)  that  would  be  included  in 
the  model  if  they  were  estimateable,  but  the  pattern  of  missing 
observations  in  the  data  base  is  a limiting  factor  (this  will  be 
discussed  in  the  data  problems  section) . There  is  still  some 
flexibility  left  so  that  after  the  initial  model  is  fitted  to  the 
data,  revisions  can  be  made  which  drop  variables  or  variable 
interactions  having  little  impact  on  the  POD. 


4.1.5  Error  Determination 


If  all  the  NDE  variables  are  fixed  at  some  level  and  the 
POD  is  observed  repeatedly,  then  the  scatter  in  the  data  must 
be  due  to  error  variability  (recall  the  error  variance  is  denoted 
by  a- 2 in  the  model),  which  may  be  thought  of  as  the  "noise" 
level  in  the  data.  However,  in  the  present  data  base  this 
information  is  not  available.  If  the  higher  order  interactions 
do  not  exist,  the  amount  of  variability  attributed  to  them 
(in  an  analysis  of  variance  summary)  is  really  error  variability. 
In  the  present  data  all  three-factor  and  higher  order  inter- 
actions were  pooled  together  and  used  as  an  estimate  of  o-2. 

' . 

The  optimal  method  of  estimating  a- 2 cannot  be  used  because 
there  are  no  repeated  observations  of  POD  under  identical  test 
conditions.  The  best  alternative  is  to  pool  higher-order  inter- 
actions to  estimate  <r  2.  The  decision  to  pool  all  three-factor 
or  higher  interactions  is  made  from  practical  considerations. 
First,  the  pattern  of  observations  is  irregular  with  missing 
combinations  of  the  factors,  some  three-factor  interactions 
cannot  be  estimated  at  all.  Secondly,  there  is  a limit  on  the 
number  of  coefficients  that  the  computer  can  estimate,  the 
inclusion  of  three-factor  or  higher  interactions  increases 
dramatically  the  number  of  coefficients  that  must  be  estimated. 
The  result  is  that  main  effects  and  two- factor  interactions  would 
have  to  be  sacrificed  in  order  to  estimate  the  higher  order 
interactions.  Lastly,  the  inclusion  of  three-factor  or  higher 
order  interactions  would  increase  the  complexity  of  the  tables 
of  coefficients  and  some  loss  of  clarity  would  surely  result. 

When  one  examines  the  values  predicted  by  a regression 
equation  there  is  another  type  of  error  that  must  be  considered, 
error  in  estimating  the  regression  coefficients.  Knowledge  of 
the  magnitude  of  this  error  allows  the  computation  of  confidence 
or  prediction  bounds  to  be  placed  on  the  estimated  regression 
equation.  In  the  regression  model  all  the  information  about  the 
variances  and  covariances  of  the  estimated  coefficients  is  con- 
tained in  the  covariance  matrix  of  fi  , which  is  (X'X)-1  rr 2.  it 
is  to  be  noted  that  the  adaptive  regression  does  not  specify  the 
model  which  is  being  fit  to  the  data  and  hence  is  not  able  to 
estimate  the  variability  due  to  error  in  estimation  of  its  re- 
gression coefficients.  This  is  a serious  weakness  because  no 
legitimate  confidence  bounds  can  be  placed  on  the  regression 
equation. 


4.2  ADAPTIVE  LEARNING  TECHNIQUES 


The  adaptive  analysis  of  the  NDE  reliability  data  consists 
of  three  phases;  a training  and  testing  set  formation,  hyner- 
surface  fitting  and  analysis.  A simplified  explanation  of  these 
three  phases  is  given  in  the  following  paragraphs.  Flow  charts, 
mathematical  equations  and  users'  instructions  for  these  computer 
programs  are  presented  in  Appendix  B. 

4.2.1  Introduction  to  Adaptive  Learning  Techniques 

Adaptive  learning  is  a mathematical  technique  for  solving 
problems  where  the  dependent  variables  depend  in  an  unknown  way 
upon  many  independent  variables  and  the  relationship  can  be  ex- 
pressed as  a multinomial  over  some  bounded  regions  of  the 
independent  variables. 

The  adaptive  learning  approach,  has  successfully  been  imple- 
mented to  predict  the  diameters  of  flat  bottom  holes(5).  The 
nonlinear  adaptive  learning  network  correctly  classified  46  out 
of  48  flat  bottom  hole  defects.  The  largest  error  in  the  classi- 
fication was  for  flat  bottom  holes  with  diameters  less  than  1/64 
inch.  The  accuracy  of  predicting  true  hole  diameter  was  97.2 
percent . 

The  adaptive  learning  network  has  been  used  to  detect  and 
measure  subsurface  fatigue  cracks (^).  The  adaptive  learning 
network  detected  and  measured  the  subsurface  fatigue  crack  in 
the  size  range  of  0 to  279  mils  to  within  70  percent  of  their 
nominally  characterized  length. 

4.2.2  Adaptive  Learning  Logic 

The  adaptive  learning  method  deals  with  high  degree  poly- 
nomials in  many  variables  and  performs  a stochastic  search  to 
evaluate  the  polynomial  coefficients  and  then  determine  the 
output  of  a given  net  connectivity  pattern. 

The  data  base  consisted  of  46,369  data  points  with  each 
point  defined  as  an  individual  inspection  performed  by  an 
operator  with  the  associated  NDT  parameters  such  as  environ- 
ment, specimen  finish,  or  crack  length.  It  will  be  noted  that 
crack  length  was  treated  as  an  independent  parameter  in  the 
analysis.  Depending  upon  the  purpose  of  the  analysis,  the  data 
points  were  divided  into  groups  with  common  NDT  parameters 
(or  input  variables).  Each  group  of  data  points  was  defined  as 
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a vector.  Associated  with  each  vector  was  a point  estimate  of 
the  detectability  which  was  obtained  by  dividing  the  number  of 
detections  by  the  total  number  of  inspections.  No  confidence 
level  was  involved  in  the  point  estimate.  If  the  crack  length 
(CL) , specimen  history  (SH) , surface  finish  (SF)  and  specimen 
thickness  (TK)  were  selected  as  the  common  input  variables,  then 
the  point  estimate  (yt)  associated  with  the  vector  can  be 
expressed  as: 


yt  = f(CLlf  SH. , SFk,  TK,  ) 

where  i,  j,  k,  ( were  subscripts  signifying  the  particular  range 
of  each  NDT  parameter.  The  data  base  was  divided  into  a training 
set  and  a testing  set  each  having  an  equal  number  of  vectors.  It 
would  have  been  desirable  to  divide  the  data  base  into  three  sets, 
and  use  the  third  set  for  evaluation.  However,  in  most  cases, 
the  data  base  was  not  large  enough  to  be  divided  into  three  sets 
and  still  retain  enough  population. 

Adaptive  learning  methodology  establishes  a polynomial  with 
estimated  coefficients  which  will  fit  the  data.  The  form  of  the 
polynomial  is  established  from  the  basic  connectivity  net  pattern 
and  the  form  of  polynomial  selected  for  the  basic  net  input. 

The  NDT  parameters  were  grouped  into  pairs  and  each  pair 
was  fed  into  a basic  net  in  a predetermined  order.  The  output 
from  the  basic  nets  was  one  of  two  types  of  polynomials  as 
indicated  below: 

A)  y = Aq  + A1X1  + A2X2  + AgX^  + A^2  + A^2 

B)  y = Aq  + A1X1  + A2X2  + A^X,, 

where  y is  the  output  of  the  basic  net,  X^  and  X2  are  input 
parameters  and  the  A's  are  coefficients.  For  most  of  the  NDE 
data  the  complete  multinomial  of  degree  two  was  used.  This 
polynomial  when  compared  to  the  linear  polynomial  seemed  to 
better  fit  the  data.  A simple  example  of  a basic  net  using  the 
NDE  parameters  of  crack  length  (CL)  and  surface  finish  (SF)  is 
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shown  below: 


y = Aq  + AXCL  + A2SF  + A3CIjSF  + A4(CL)2  + A5(SF)2. 


It  should  be  noted  that  the  predicted  point  estimate  is  obtained 
from  the  output  of  the  complete  network.  The  basic  connectivity 
net  pattern  used  in  this  program  was  a rectangular  network  with 
exponential  pitch.  The  number  of  rows  is  equal  to  the  number  of 
input  parameter  pairs.  The  number  of  columns  were  optional. 


There  are  three  major  types  of  feed  forward  nets: 
rectangular,  triangular,  and  exponential.  The  effective- 
ness of  the  three  nets  is  about  the  same  for  a given  class 
of  problems,  and  the  decision  as  to  which  net  to  use  can 
usually  be  made  on  the  grounds  of  programming  ease  or  computer 
running  time.  Since  the  smallest  possible  net  for  a given  number 
of  inputs  is  an  exponential  net,  this  net  is  the  most  logical 
choice  for  a first  analysis  of  a problem.  Since  the  rectangular 
net  can  be  made  with  an  arbitrary  number  of  columns  and  since  the 
degree  of  the  highest  power  terms  in  the  multinomial  is  roughly 
proportional  to  the  number  of  columns  in  a net,  the  rectangular 
net  is  also  a useful  net  for  many  problems. 

The  basis  of  the  fitting  program  was  a stochastic  search 
in  which  the  polynomial  coefficients  were  changed.  The  pre- 
dictive point  estimate  was  calculated  for  each  set  of  coefficient 
and  compared  to  the  true  average  point  estimate  of  the  vectors. 
The  comparison  was  quantified  by  a score  which  is  defined  as 
follows : 


SCORE 


where  N is  the  number  of  vectors,  y^  and  y are  the  true 
point  estimate  and  the  predictive  point  estimate  of  the 
detectability  respectively. 


Initially  a score  was  obtained  from  the  training  set 
vectors  by  using  a random  set  of  coefficients  (BTDTR),  where 
N is  the  number  of  vectors.  After  obtaining  initial  scores 
the  iterative  process  of  determining  coefficents  for  the  best 
fit  to  the  data  is  started.  On  the  first  trial  the  search 
resorts  to  an  unguided  search.  Unguided  refers  to  the  se- 
lection of  a set  of  bounded  random  coefficients  (ACCUR) . After 
going  through  the  unguided  phase  a programmable  integer  (N) 
option  is  selected  so  that  the  search  will  not  be  permitted  to 
go  unguided  until  after  N trials.  This  is  referred  to  as 
embedding.  Using  ACCUR  a score  is  computed  for  the  training 
set,  and  a new  set  of  coefficients  is  computed  as  follows: 

ADELT  = ACCUR- BTDTR. 

The  search  will  either  branch  to  the  training-testing  paradigm 
or  the  reversal  phase  described  below  depending  on  whether  the 
score  has  improved. 

The  reversal  phase  determines  a new  set  of  coefficients 
as  follows: 

ACCUR  = BTDTR -ADELT. 


ACCUR  is  checked  to  insure  that  the  coefficients  do  not  exceed 
the  bounded  region.  The  reversal  phase  uses  ACCUR  to  calculate 
the  training  set  score.  Depending  on  the  score  the  search  will 
either  branch  to  the  training-testing  paradigm  or  the  guided 
phase  of  the  search. 

In  the  guided  phase  a new  set  of  coefficients  (DELT)  is 
determined  from  scaled  random  numbers.  These  coefficients  are 
used  to  determine  an  additional  set  as  follows: 

ACCUR  = BTDTR  + DELT. 

The  set  of  coefficients  (ACCUR)  is  checked  to  insure  that  the 
coefficients  do  not  exceed  the  bounded  region.  A score  for  the 
training  set  is  computed  using  ACCUR.  The  search  will  either 
branch  to  the  training-testing  paradigm  or  reversal  depending 
on  whether  the  score  has  been  improved.  Any  time  the  training 
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set  score  is  improved,  the  search  branches  to  the  training- 
testing paradigm,  and  the  training  set  coefficients  are  updated 
as  follows: 


BTDTR  = ACCUR. 

The  set  of  coefficients  (BTDTR)  is  used  in  conjunction  with 
the  test  set  vectors  to  determine  the  score  of  the  testing 
set.  If  the  test  set  score  is  improved,  this  set  of  coefficients 
is  retained  and  the  search  branches  to  the  acceleration  phase. 

In  the  acceleration  phase  a new  set  of  coefficients  is 
obtained  as  follows: 


ADELT  = 2*DELT. 


A set  of  coefficients  is  then  determined  by  using  ADELT  in 
conjunction  with  BTDTR  as  follows : 

ACCUR  = ADELT  + BTDTR. 


The  set  of  coefficients  (ACCUR)  is  used  to  determine  the  score 
of  the  training  set.  If  the  score  represents  an  improvement  the 
search  branches  to  the  training-testing  paradigm.  If  no  improve- 
ment in  score  is  obtained  the  search  will  branch  to  the  guided 
phase.  The  search  continues  in  this  iterative  manner  described 
above  until  the  test  set  score  has  been  reduced  to  a minimum. 

In  the  final  phase  of  analysis,  a parametric  study  is 
performed.  Results  from  this  study  are  in  the  form  of  graphical 
plots  which  show  the  predicted  point  estimate  as  a function  of 
crack  length.  This  calculation  can  only  be  made  providing  the 
remaining  input  parameters  are  fixed.  If  all  the  coefficients  of 
a net  are  known  and  all  the  other  variables  are  fixed  except 
crack  length,  then  it  is  possible  to  calculate  the  predicted  point 
estimate  as  a function  of  crack  length. 
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SECTION  V 


MODEL  DEVELOPMENT 


The  linear  regression  techniques  and  the  adaptive  learning 
techniques  described  in  the  previous  section  were  used  to  de- 
velop the  translation  model.  A successful  model  takes  into 
consideration  all  the  pertinent  parameters  that  influence  the 
inspection  results.  The  inspection  results  in  a NDE  facility 
are  expressed  in  terms  of  probability  of  detection  for  some 
defect  size  in  a material.  The  flaw  detection  capability  was 
related  to  each  of  the  pertinent  NDE  parameters  by  a parametric 
study.  Effects  on  the  inspection  results  from  a combination  of 
the  parameters  have  been  included  in  the  model  development. 

The  procedures  for  the  translation  model  development  using 
the  two  analytical  techniques  are  presented  in  this  section. 

The  presentation  will  be  conducted  in  the  form  of  examples  to 
illustrate  the  development  process.  Problems  encountered  during 
the  development  for  each  technique  will  be  discussed  and  a com- 
parison of  the  pros  and  cons  of  the  two  techniques  will  be  made. 
The  results  of  the  parametric  study  and  translation  model  de- 
velopment will  be  postponed  until  the  next  section. 

5.1  LINEAR  REGRESSION  TECHNIQUE  IN  MODEL  DEVELOPMENT 

The  model  development  procedures,  analysis  capabilities, 
data  problems  encountered  in  the  analysis, and  some  general 
comments  will  be  presented  in  the  following  paragraphs. 

5.1.1  Translation  Model  Development 

The  purpose  of  the  translation  model  is  to  compare  the  PODs 
for  flat  plate  specimens  with  the  PODs  for  specimens  with  more 
complex  geometries.  There  are  two  basic  methods  that  can  be 
used  to  make  this  comparison,  the  ratio  method  and  the  difference 
method.  For  comparison  purposes  assume  that  the  POD  has  been 
modeled  as  a function  of  geometry,  crack  length,  surface  finish 
and  operator  in  the  following  way: 

*ijkf  ■',  + °i+^+>'k+  •(  + + (^y)Jk 
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where 


= POD  observed  with  geometry  i, 

crack  length  j, 
surface  finish  k, 
operator  ( , 


and 


U 


- constant. 


a . 
1 


geometry  effect  (i  * 1 is  flat  plate, i 
is  integrally  stiffened  panel  (ISP)), 


= 2 


= crack  length  effect. 


Fk  = surface  finish  effect, 

= operator  effect. 


(a  & )ij  = geometry  by  crack  length  interaction, 

(/?  y )jk  = crack  length  by  surface  finish  interaction. 

For  purpose  of  simplicity  in  this  illustrative  example  some  inter- 
ation  terms  have  been  deleted.  The  ratio  of  POD  on  flat  plate 
specimens  to  the  POD  on  integrally  stiffened  panel  specimens  is 
then: 


Y1  jkf  = + yk+  6t  + + (0Y  )jk 

Y2  jkf  /y+a2+^j+yk+^  + + Uy)jk 

(Notice  that  this  ratio  is  a function  of  crack  length,  surface 
finish  and  operator.) 


The  difference  between  the  POD  on  flat  plate  specimens  and 
the  POD  on  integrally  stiffened  panel  specimens  is: 

Y1  jkf  “ y2  jkf  = “ 1 " a2  + >lj  - <“0  >2j  • 

Notice  the  difference  is  a function  of  only  crack  length,  which 
means  the  inference  is  much  broader.  Another  important  point, 
suppose  there  is  a large  difference  between  the  PODs  on  flat 
plate  and  stiffened  panel  specimens,  the  numerator  and  denominator 
of  the  ratio  contain  several  common  terms  which  can  obscure  the 
difference.  The  difference  does  not  have  these  common  factors 
and  is  thus  much  more  sensitive  to  differences  in  PODs.  When  a 
translation  model  is  developed  it  is  desirable  to  be  able  to 
estimate  the  extent  of  error  in  the  model.  The  statistical  dis- 
tribution of  the  ratio  type  model  is  very  difficult  to  obtain 
but  the  distribution  of  the  difference  type  model  is  much  easier. 
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Most  of  the  information  to  compute  confidence  bounds  or  prediction 
bounds  on  the  difference  in  PODs  is  available  in  the  Statistical 
Analysis  System  program  package  currently  in  use  at  different 
statistical  analysis  centers. 

The  only  terms  in  the  difference  type  model  that  do  not 
cancel  during  subtraction  are  the  main  effects  due  to  geometry 
differences  and  interactions  of  variables  with  geometry.  The 
implication  is  that  special  attention  should  be  given  to  variable 
interactions  with  geometry.  The  actual  model's  fit  with  the  cor- 
responding translation  model  is  detailed  in  the  example  given  in 
the  following  section. 


5.1.2  Analysis  Capabilities 

The  total  amount  of  variability  in  the  observations  of  the 
POD  can  be  measured  quantitatively  by  the  sum  of  squared  devia- 
tions from  the  mean  POD.  Each  variable  or  interaction  of  vari- 
ables will  explain  part  of  the  total  variability,  the  larger  the 
fraction  of  the  variability  explained  (quantitatively  expressed 
in  terms  of  sums  of  squared  deviations)  the  larger  the  impact  on 
the  POD.  It  is  most  desirable  to  partition  the  total  variability 
into  disjoint  parts,  with  each  part  being  unambiguously  associated 
with  an  independent  variable.  However  this  is  not  possible  in 
the  present  data  set  because  of  the  pattern  of  missing  data  cells. 
The  next  best  solution  is  to  sequentially  partition  the  total 
variability.  This  means  that  one  variable  is  selected  and  the 
amount  of  the  total  variability  (sum  of  squares)  that  it  accounts 
for  is  computed.  Another  variable  or  interaction  of  variables  is 
then  selected  ai^d  the  amount  of  variability,  which  remained  after 
the  first  variable  accounted  for  its  portion,  is  computed.  The 
process  continues  with  each  successive  variable  being  given  a 
chance  to  account  for  variability  left  over  from  the  previous 
variables.  This  process  is  obviously  "order  of  variable  selec- 
tion" dependent,  hence  the  most  important  variables  (physically 
meaningful  in  the  opinion  of  the  NDE  personnel)  should  be  in- 
cluded first  in  this  sequential  partition.  Fortunately  the  values 
of  the  coefficients  estimated  are  independent  of  order. 

In  analyzing  whether  the  model  which  was  fitted  is  adequate, 
the  original  observed  values  of  the  POD  along  with  the  values 
predicted  by  the  model  are  printed  out.  Patterns  in  the  resi- 
duals can  be  used  to  revise  the  model  if  necessary. 

If  one  is  willing  to  assume  normality  of  errors  then  tests 
of  hypothesis  about  whether  a particular  variable  is  accounting 
for  a significant  amount  of  variability  can  be  performed.  Con- 
fidence or  prediction  intervals  can  be  computed  to  quantify  the 
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amount  of  error  that  should  be  associated  with  the  regression 
equation  (POD  model).  As  mentioned  in  the  error  determination 
section,  higher  order  interactions  are  pooled  together  to  provide 
an  estimate  of  error  variability.  The  indications  are  good  that 
proper  selections  were  made  of  interactions  that  were  pooled  for 
"error",  because  the  amount  of  variability  accounted  for  per 
degree  of  freedom  is  6.6  times  higher  for  the  terms  in  the  model 
than  for  the  "error"  terms.  In  the  sample  that  follows,  the 
coefficients  needed  for  a difference  or  ratio  translation  model 
are  presented.  In  some  cases  the  estimated  coefficients  are  0 
due  to  lack  of  observations  in  critical  data  cells  (this  problem 
and  related  data  problems  are  discussed  in  the  next  section).  It 
is  also  to  be  noted  that  the  notation  is  slightly  different,  i.e., 
(CL)^  is  used  to  represent  crack  length  effect,  etc. 

The  data  to  be  discussed  in  this  sample  relate  to  flat  plate 
and  integrally  stiffened  panel  specimens,  both  ultrasonically  in- 
spected. There  are  a total  of  259  observed  PODs , hence  258  degrees 
of  freedom  are  associated  with  the  observations.  The  total  amount 
of  variability  in  the  POD  (dependent  variable)  as  measured  by  the 
sum  of  squared  deviations  from  the  mean  is  17.94.  In  the  example 
below,  65  coefficients  were  estimated.  (See  source  of  variation 
table  below.  There  is  one  degree  of  freedom  for  every  estimated 
coefficient).  By  estimating  65  regression  coefficients,  it  is 
possible  to  account  for  12.37  of  the  total  17.94  sum  of  squares, 
which  is  about  69  percent  of  the  total  variability.  This 
information  can  be  summarized  in  an  "analysis  of  variance"  table, 
as  follows. 


Source  of  Variation 

DF 

SS 

MS 

Total  (corrected) 

258 

17.94 

Regression  Coefficients 

65 

12.37 

.190 

"ERROR" 

193 

5.57 

.029 

where 


DF  = Degrees  of  Freedom 

SS  = Sum  of  Squares 

MS  = Mean  Square  = SS/DF. 


Detailed  Breakdown  of  Source 

of  Variation  for  Regression 

Coefficients 

DF 

SS 

Crack  Length 

8 

7.80 

Operator 

5 

1.07 

Geometry 

1 

.16 

Crack  Length  by 

Geometry  Interaction 

5 

.68 
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Detailed  Breakdown  of  Source 
of  Variation  for  Regression 
Coefficients  (Cont'd) 


Source  of  Variation 

DF 

SS 

Crack  Length  by 

Thickness  interaction 

15 

.47 

Crack  Length  by 

Surface  finish  interaction 

25 

2.03 

Surface  Finish  by 

Geometry  interaction 

1 

.04 

Specimen  History  by 

Geometry  interaction 

2 

.04 

Surface  Finish  by 

Thickness  interaction 

3 

.08 

Total 

65 

12.37 

The  linear  model  for  this  data  can  be  written  in  an  abbre- 
viated form  as: 

P0Dijkfmn  = **  + (CL)i  + (0PID)j  + (Geom)k  + (CL  *Geom)ik 
+ (CL  * TK)ii?  + (CL*SF)im  + (SF’^Geom),^  + (Hist  * Geom)nk 
+ (SF*TK)mi,  + error 

where 

CL  = Crack  Length 
OPID  = Operator  ID 
Geom  = Specimen  Geometry 
TK  = Thickness 
SF  = Surface  Finish 
Hist  = Specimen  History 

* = Symbol  indicating  variable  interaction  effect 
4 = constant 

The  coefficients  for  this  model  follow: 

4 = .826 


1 

-.698 

OPID  0 

.064 

Geom  1 

.106 

2 

-.733 

15 

-.025 

2 

0 

3 

-.193 

16 

.070 

4 

-.801 

17 

-.064 

5 

-.053 

18 

.028 

6 

.019 

19 

0 

7 

-.095 

8 

.081 

9 

0 

36 
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The  information  above  can  be  summarized  in  the  following 
tables  of  differences: 


(Geom)^  - (Geom^  = .106 
(CL  * Geom)^  - (CL  * Geom)^ 


i 


0 

0 

0 

.656 

.019 

.019 

.008 

.071 

0 


(SF  * Geom)^^  - (SF  * Geom)^ 


m = 0 
1 
2 

3 

4 

5 


0 

.063 

0 

0 

0 

0 


(Hist  * Geom)^  - (Hist  * Geom)^ 


n = 1 
2 


-.041 

0 


It  will  be  noted  that  the  main  effect  of  thickness  had 
virtually  no  impact  on  the  POD  and  was  dropped  from  the  model, 
hence  no  table  was  necessary.  Furthermore,  the  interactions  of 
thickness  by  geometry,  specimen  history  and  operator  ID  were 
also  found  to  be  insignificant  and  were  dropped  from  the  model. 
However,  only  the  thickness  by  crack  length  interaction  and 
thickness  by  surface  finish  contributions  were  more  sizeable 
than  the  main  effect  of  the  specimen  thickness.  These  two 
interaction  terms  were  retained  in  the  model. 
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An  example  is  given  for  using  the  tables  of  coefficients 
to  estimate  the  POD.  Suppose  it  is  desired  to  predict  the 
POD  on  a specimen  whose  CL  = 5,  Geom  = (flat  plate).  TK  = 1, 
SF  = 2,  Hist  = 1,  and  OPID  = 15.  Begin  with  the  constant  U = 
.826.  Correct  this  value  additively  for  each  variable  and 
variable  interaction  on  the  model  given  on  page  36: 


(CL)5 

= 

- .053 

(opid)15 

= 

- .025 

(Geom) ^ 

= 

.106 

(CL*Geom),-  ^ 

= 

.019 

(cl*tk)5j1 

= 

.020 

(cl*sf)5>2 

= 

- .004 

(SF*Geom)2 

= 

0 

(Hist*Geom) ^ ^ 

= 

- .077 

(sf*tk)2  ’ 

= 

.015 

Totaling  the  corrections  yields 

+ .001,  added  to  U = .826  gives 

us  .827  as  the  estimated  POD 

under  the  conditions  stated  above. 

The  POD  under  the  same 

conditions  except  that  the  specimen 

geometry  is  integrally  stiffened 

panel  rather  than  flat  plate  is 

found  in  the  same  manner  as 

for 

flat  plate  specimens: 

Flat  Plate  (Geom  = 1) 

ISP  (Geom  = 2) 

H = .826 

H = .826 

(CL)5  = - .053 

(CL)5  = - .053 

(OPID)15  = - .025 

(0PID)15  = - .025 

(Geom)^  = .106  ^ 

(Geom)2  = 0 

(CL*Geom)5  l = .019 

(CL*Geom)t-  2 = 0 

(CL*TK)5  1*=  .020 

(CL*TK)5  1 = .020 

(CL*SF)5’2  = - .004 

(CL*SF)5,2  = - .004 

(SF*Geom)2  ^ = 0 

(SF*Geom)2  = 0 

(Hist*Geom)^  ^ = - .077 

(Hist*Geom)^  2 = - .036 

(SF*TK)2  x = .015 

(SF*TK)2  1 =* .015 

Looking  down  the  two  lists  of  corrections,  it  is  seen  that  the 
only  terms  that  change  are  terms  involving  geometry.  Therefore 
to  find  the  difference  in  POD  on  flat  plate  specimens  compared 
to  integrally  stiffened  panel  specimens  all  the  tables  on  page 
37  are  not  needed.  From  the  tables  of  differences  on  page  33 
the  differences  are: 

(Geom) ^ - (Geom)^  = .106 

(CL*Geom)<-  ^ - (CL*Geom)^  2 = .019 
(SF*Geom)2  l - (SF*Geom>2  2 = 0 
(Hist*Geom)^  ^ - (Hist*Geom)^  2=  ".041 


Total  .084 

The  difference  between  the  POD  for  flat  plate  (.827)  and  the  POD 
for  ISP  (.743)  is  .084.  A comment  is  appropriate  here  about 
additional  testing  which  would  improve  the  precision  of  the 
estimates  derived  from  the  data.  Since  variables  interacting 
with  geometry  determine  the  difference  in  POD,  maximum  information 
on  these  interactions  is  desirable.  The  two-factor  interactions 
can  be  estimated  adequately  if  data  for  all  combinations  of 
geometries  with  the  other  variables  are  available  for  analysis. 
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5.1.3  Data  Problems 


The  most  difficult  types  of  data  to  be  analyzed  statistically 
are  large  sets  of  data  with  many  variables  represented  along  with 
an  irregular  pattern  of  missing  data  cells.  The  first  problem 
is  with  confounded  variable  effects.  This  problem  occurs  when 
the  effects  of  variables  are  being  isolated.  For  example,  sup- 
pose there  are  9 possible  combinations  of  surface  finish  and 
thickness  but  only  3 observed  PODs  as  follows: 


Surface  Finish 
12  3 


(M  * missing) 


<u 

2 

u 

•H 

-C  cv| 
H 


on 


Y 

Y11 

M 

M 

M 

Y22 

M 

M 

M 

Y33 

When  the  difference  between  the  Y's  (PODs)  is  being  explained,  it 
is  impossible  to  determine  whether  the  differences  are  due  to 
changes  in  surface  finish  or  are  due  to  changes  in  thickness. 

This  is  a rather  obvious  case  of  complete  variable  confounding. 
There  are  many  more  subtle  cases  of  confounding  in  the  data. 

These  problems  may  be  identified  at  least  partially  by  the  se- 
quential partition  of  the  total  sum  of  squares.  The  amount  of 
variability  accounted  for  by  thickness,  after  surface  finish 
has  accounted  for  its  portion,  is  zero.  Likewise  the  amount  of 
variability  accounted  for  by  surface  finish,  after  thickness  has 
accounted  for  its  portion,  is  also  zero. 

When  the  confounding  is  not  complete  as  in  the  previous 
example  there  may  be  a partial  overlapping  of  variable  effects, 
the  "partial  sum  of  squares"  is  the  amount  of  variability  ac- 
counted for  by  a variable  after  all  the  other  variables  have  had 
an  opportunity  to  account  for  variability. 

Another  problem  can  be  illustrated  with  the  same  example 
above,  namely  the  problem  of  nonestimateable  variable  effects 
caused  by  missing  data  cells.  In  the  example  above,  it  may  be 
desirable  to  estimate  thickness  effects,  surface  finish  effects 
and  thickness  by  surface  finish  interaction  effects.  However, 
there  is  not  enough  data  to  estimate  all  these  effects.  It  is 
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important  to  understand  that  a number  could  be  associated  with 
each  of  these  effects  but  there  would  be  absolutely  no  confidence 
in  the  result. 

All  of  the  analysis  problems  occur  because  of  missing  data 
cells  which  result  in  irregular  data  patterns.  The  present  study 
has  provided  indications  of  which  variables  are  most  significant. 
If  the  list  of  variables  could  be  narrowed  to  the  point  where  it 
would  be  feasible  to  fill  in  all  or  most  of  the  missing  variable 
combinations,  the  quality  of  the  resulting  analysis  would  be 
improved  immeasurably. 


5.1.4  General  Comparative  Comments 

The  linear  regression  approach  is  to  specify  the  model  that 
is  to  be  fitted  to  the  data  and  then  to  estimate  the  unknown  coef- 
ficients in  the  model  (as  outlined  in  a previous  section) . The 
coefficients  are  not  constrained.  If  a model  is  specified  with 
more  than  200  coefficients,  the  present  computing  capability  is 
exceeded.  This  is  the  only  motivation  for  the  adaptive  approach 
to  regression.  The  adaptive  approach  uses  a random  search  method 
to  establish  coefficients  for  terms  in  a polynomial.  The  coef- 
ficients are  functionally  related,  the  exact  form  of  the  dependency 
is  determined  by  the  "net"  used  to  generate  the  polynomial.  If 
a net  can  be  found  which  induces  constraints  which  are  compatible 
with  the  data,  a reasonably  good  fit  may  result. 

As  was  pointed  out  previously,  the  coefficients  in  a poly- 
nomial regression  model  are  difficult  to  use  directly  to  compare 
the  effects  of  NDE  variables.  This  problem  is  compounded  by  the 
artificial  constraints  placed  on  the  coefficients  by  the  adaptive 
procedure.  Since  no  model  is  specified,  and  since  there  are  con- 
straints among  the  coefficients,  it  is  impossible  to  determine 
how  each  estimated  coefficient  is  derived  from  the  data.  There- 
fore it  is  impossible  to  estimate  the  error  that  is  being  made 
in  estimating  the  coefficients.  It  also  follows  that  confidence 
intervals  or  prediction  intervals  cannot  legitimately  be  placed 
on  the  estimated  model.  The  adaptive  procedure  uses  the  "train- 
ing-testing paradigm"  to  avoid  "overfitting"  the  data.  The  pro- 
perties of  this  procedure  have  not  been  defined,  especially  for 
cases  where  many  data  cells  are  missing.  Further  development  of 
the  adaptive  learning  procedures  is  needed  for  future  incorporation 
into  the  standard  linear  regression  procedures. 
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SECTION  VI 
RESULTS 


The  results  of  the  parametric  study  and  the  translation 
model  development  are  presented  in  this  section.  These  results 
were  obtained  by  using  the  statistical  analysis  schemes  described 
in  the  two  previous  sections.  The  parametric  relationship  and 
the  translation  model  established  in  this  program  were  the  key- 
stones for  the  design  of  an  optimum  demonstration  program.  The 
translation  model  also  makes  the  interpretation  of  the  capabili- 
ties of  a facility  more  realistic. 

6.1  NDE  PARAMETER  STUDY 

The  successful  development  of  a demonstration  program  to 
assess  the  flaw  detection  capability  of  a NDE  facility  is  con- 
tingent on  a sound  knowledge  of  the  parameters  that  may  influence 
inspection  results  and  their  degree  of  influence.  Based  on  the 
data  compiled  in  the  computer  data  bank,  the  identification  of 
pertinent  NDE  parameters  and  a study  of  the  parametric  relation- 
ships were  successfully  completed. 

6.1.1  Parameter  Identification 

A comprehensive  list  of  pertinent  parameters  was  compiled 
for  each  NDE  technique  considered  in  this  program.  The  para- 
meters are  separated  into  two  general  categories:  (1)  those 
that  are  common  to  all  techniques  and  (2)  those  that  apply  speci- 
fically to  one  technique.  A cover  sheet  preceding  each  data  set 
in  the  computer  data  bank  contains  information  about  the  para- 
meters pertinent  to  that  data  set.  Each  parameter  is  identified 
by  a computer  code  with  its  associated  description  to  facilitate 
sorting  of  parameters  in  parametric  studies.  A list  of  the  rele- 
vant NDT  parameters  for  each  of  the  five  techniques,  ultrasonic, 
eddy  current,  X-ray,  magnetic  particle,  and  liquid  penetrant, 
was  presented  in  Table  2 earlier  and  in  Tables  A-l  to  A-4  in  Appendix  A. 

6.1.2  Parametric  Relationships 

The  parametric  relationship  between  the  inspection  results 
and  the  pertinent  NDE  parameters  was  studied  by  three  different 
analytical  methods:  (1)  optimum  probability  method  (OPM), 
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(2)  point  estimate,  and  (3)  statistical  fitting  schemes.  A sub- 
jective comparison  of  different  statistical  schemes  indicated 
that  the  OPM  scheme  appears  to  be  the  best  procedure  among  the 
statistical  procedures  discussed  in  Reference  1.  However,  this 
scheme  is  not  considered  appropriate  for  the  purpose  of  comparing 
the  influence  of  different  parameters  on  POD  due  to  the  inequality 
in  sample  size.  Therefore,  the  point  estimate  and  the  statistical 
fitting  schemes  were  selected  for  the  parametric  relationships  study 

6. 1.2.1  Comparison  of  Statistical  Evaluation  Schemes 

A detailed  discussion  of  the  pros  and  cons  of  the  range, 
overlapping  60  points,  and  OPM  schemes  for  calculating  the  PODs 
is  presented  in  Appendix  C.  The  results  of  this  subjective 
evaluation  indicated  that  the  OPM  scheme  is  the  best  procedure 
because  it  takes  into  consideration  the  problem  of  unequal  sample 
size  and  makes  full  use  of  the  available  data.  The  disadvantage  of 
this  scheme  is  the  liberal  bias  in  crack  size  regions  where  the  POD 
curve  is  either  flat  or  increasing  slowly. 

Qualitative  comparisons  of  the  influences  of  NDE  parameters 
on  the  inspection  capabilities  can  be  obtained  by  comparing  the 
POD  plots  as  a function  of  crack  length.  In  these  comparisons, 
the  detection  threshold  (D.T.)  is  defined  as  a minimum  detect- 
able crack  length  with  a POD  above  0.90  at  95  percent  confidence 
level.  Figure  1 presents  a comparison  of  the  POD  difference  due 
to  surface  finish  in  integrally  stiffened  panel  specimens  for 
eddy  current  technique.  This  figure  represents  an  example  of 
the  POD  curves  available  through  the  computer  retrieval  and  analy- 
sis system  of  the  data  bank  of  222  data  sets.  These  POD  curves 
provide  a valid  comparison  of  the  influences  of  NDE  parameters 
from  a NDE  demonstration  program  standpoint  since  MIL-A-83444 
requires  that  a minimum  detectable  crack  size  be  determined  in 
terms  of  0.90  POD  at  95  percent  confidence  level.  Although 
these  criteria  may  not  be  appropriate  from  the  standpoint  of 
parametric  relationship  study,  they  will  certainly  be  instrumen- 
tal in  the  application  of  the  parametric  relationship  towards 
the  translation  model  development. 

6. 1.2. 2 Point  Estimate  of  POD 

In  the  point  estimate  comparison,  the  inspection  results 
for  each  technique  were  grouped  in  1/8  in.  crack  length  intervals 
and  plotted  in  the  form  of  histograms  for  the  POD.  Analysis  of 
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the  histogram  data  provided  a good  indication  of  the  order  of 
importance  of  the  influencing  parameters  on  the  POD  and  indi- 
cated data  deficiencies  in  some  areas.  The  NDE  reliability 
data  base  is  such  that  not  all  the  NDE  parameters  can  be  analyzed 
by  linear  regression  analysis  or  adaptive  learning  techniques  due 
to  insufficient  number  of  inspections  in  most  cases.  Table  4 
presents  a list  of  the  parameters  that  were  included  in  the  point 
estimate  comparison  and  linear  regression  analysis.  Figure  2 
shows  sketches  of  some  specimen  geometries  listed  in  Table  4. 

Some  of  the  specimens  were  inspected  with  as -machined  surfaces 
and  then  re-inspected  after  etching  to  determine  the  effect  of 
surface  treatment  on  the  POD.  Cracks  in  the  form  of  compressed 
notches  were  present  on  the  flat  portions  of  the  tandem  T 
specimens'?' . These  compressed  notch  cracks  were  fabricated  by 
grinding  grooves  of  0.020  in.  radius  in  the  specimen  blanks  and 
placing  EDM  notches  of  0.005  to  0.008  in.  in  the  grooves.  The 
specimens  were  then  subjected  to  axial  compression  loading  of 
sufficient  magnitude  to  close  the  notches  by  plastic  deformation. 
Following  compression,  the  grooves  were  machined  to  attain  the 
final  configuration.  Typical  crack  opening  of  the  compressed 
notch  cracks  in  aluminum  was  approximately  0.0002  in.  Compressed 
notch  cracks  on  the  inner  and  external  surfaces  of  straight  hollow 
or  solid  cylinders  can  be  similarly  fabricated.  These  specimen 
geometries  were  not  illustrated  in  Figure  2.  The  weld  specimens 
had  crowns  (weld  beads)  in  the  as-welded  condition.  They  were 
inspected  in  that  configuration  and  then  re-inspected  after  the 
beads  were  machined  off  until  the  surface  in  the  weld  zone  was 
flush  with  the  surface  of  the  parent  material.  The  machining 
operation  was  termed  scarfing  in  the  specimen  fabrication  process' 

A computer  program  was  coded  to  divide  the  specimen  crack 
length  into  eight  equal  intervals  of  0.125  in.  in  each  data  set. 

The  number  of  detections  in  each  crack  length  interval  was  divided 
by  the  number  of  inspections  to  obtain  the  point  estimate.  Any 
of  the  parameters  listed  in  Table  4 could  be  compared  to  any  other. 
The  comparison  is  conducted  by  obtaining  the  ratio  of  the  point 
estimates  of  the  PODs  corresponding  to  the  two  parameters.  An 
example  is  given  in  Figure  3 to  show  one  of  the  parametric  relation- 
ships presented  in  a computer-printed  histogram  form.  The  NDE 
method  for  the  histogram  is  printed  as  a heading  followed  by  the 
data  set  numbers  from  which  the  data  have  been  obtained.  In  some 
cases  subsets  from  the  data  sets  are  identified  by  their  parameter 
codes  such  as  surface  finish.  The  printouts  starting  from  the 
middle  of  the  second  row  identify  the  parameter  whose  relation- 
ship to  POD  is  to  be  determined.  The  date  and  time  of  the  estimates 
are  placed  at  the  extreme  right  hand  end  of  the  second  row.  Other 
illustrations  of  the  computer  generated  histograms  for  comparing  the 
effects  on  POD  of  other  NDE  parameters  are  shown  in  Appendix  D.  Only 
pertinent  conclusions  are  summarized  in  the  following  paragraphs. 
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TABLE  4 


A LIST  OF  NDE  PARAMETERS  INCLUDED 
IN  THE  PARAMETRIC  STUDY 


Category 


NDT  Method 


Code 


Eddy  Current  (EC) 

Liquid  Penetrant  (Penetrant) 

Magnetic  Particle 

X-Ray 

Ultrasonics 


Sample  History 


Data  Source 


Part  Geometry 


As  Machined  Surface 
Etched  Surface  (Etched) 

Scarfed 

Proof  Loaded  (Proof) 

Martin  Marietta/Denver  Div.  (NAS  CR-2369) 
Rockwell  International/Space  Div.  (NAS  9-14000) 
General  Dynamics/Convair  Div.  (NAS  9-14000) 
Martin  Marietta/Denver  Div.  (NAS  9-14000) 

Boeing  Commercial  Airplane  Co.  (AFML-TR- 74-241) 
Martin  Marietta/Denver  Div.  (NAS  9-13578) 

Flat  Plate  (FP) 

Integrally  Stiffened  Panel  (ISP) 

Flat  Plat  Riveted  to  Integrally  Stiffened  Panel 
Lack  of  Penetration  (LOP)  for  Welded  Panel 
Longitudinal  Welded  Panel  with  Crowns 
Transverse  " " " " 

Longitudinal  " " without  Crowns 

Tandem  T 


Operator  ID 


Operator  A 
Operator  B 


TABLE  4 (CONTINUED) 

A LIST  OF  NDE  PARAMETERS  INCLUDED 
IN  THE  PARAMETRIC  STUDY 


Category 

Code 

Key 

Inspection 

Environment 

1 

Production 

2 

Laboratory 

Material 

1 

Aluminum 

2 

Steel 

Specimen 

Finish 

1 

1-32  RMS 

2 

33-64  RMS 

3 

65-128  RMS 

4 

129-250  RMS 

5 

Larger  than  250  RMS 

Specimen 

Thickness 

1 

1-200  Mils 

2 

201-300  Mils 

3 

301-500  Mils 

a/2c 

1 

Below  0.17 

2 

0.17  - 0.34 

3 

Above  0.34 
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Figure  3 Example  of  Computer  Printouts  of  Point  Estimate  Comparisons 
in  Parametric  Relationship  Study  on  Flat  Plate  Specimens 


Specimen  surface  finish  was  observed  to  have  negligible 
effect  on  the  POD  for  ultrasonic  and  X-ray  methods.  For  eddy 
current  and  penetrant  inspections  the  effect  is  noticeable 
only  in  the  smallest  crack  length  region  of  0 - 0.125  in. 

Comparing  specimen  thicknesses,  the  only  significant  difference 
in  POD  for  specimens  with  two  different  thicknesses  was  observed 
for  the  X-ray  technique.  Etching  the  specimen  surface  was  found 
to  increase  the  POD  for  eddy  current,  penetrant  and  X-ray  tech- 
nique. No  noticeable  effect  could  be  discerned  from  the  etching 
for  the  ultrasonic  technique.  A comparison  of  the  POD  for 
different  inspectors  within  the  same  company  revealed  that  the 
inspection  efficiency  for  these  inspectors  appeared  to  be  quite 
uniform.  Similar  comparison  conducted  on  three  companies  using 
the  same  specimens  with  fatigue  cracks  showed  very  little 
variation  in  inspection  efficiency.  Comparing  specimen  geometry, 
the  POD  for  the  simpler  geometry  were  higher  than  those  for  the 
more  complex  geometry  for  all  three  NDT  techniques.  However, 
the  difference  was  significant  only  for  the  smallest  crack  length 
range  in  the  case  of  ultrasonic  inspection. 

For  NDT  techniques  of  penetrant,  eddy  current,  and  ultra- 
sonics, PODs  obtained  on  flat  plates  were  higher  than  corresponding 
values  for  the  integrally  stiffened  panels  (ISP)  and  ISP  with  a 
riveted  plate.  However,  little  difference  could  be  discerned  for 
the  integrally  stiffened  panels  with  and  without  a riveted  plate. 

A comparison  of  POD  for  weld  specimens  with  as-welded  and  scarfed 
joints  using  penetrant,  ultrasonic,  and  eddy  current  techniques 
showed  that  the  PODs  for  the  two  types  of  specimen  histories  were 
essentially  equivalent  for  the  ultrasonic  and  eddy  current  tech- 
niques. For  the  penetrant  technique,  lower  PODs  were  evident  for 
specimens  with  scarfed  joints.  This  appeared  to  be  contradictory 
to  the  expected  trend.  The  reason  for  the  anomaly  could  be 
attributed  to  a smearing  of  the  scarfed  surfaces  of  the  aluminum 
weld  specimen.  The  flaw  openings  to  the  specimen  surface  were 
closed  by  the  metal  chips  preventing  the  penetration  of  the 
penetrants.  Except  for  the  eddy  current  technique,  the  PODs 
obtained  for  weld  specimens  with  lack  of  penetration  defects  after 
a proof  loading  of  90  percent  of  the  yield  stress  were  much  higher 
compared  to  those  obtained  before  proof  loading.  The  point  estimate 
comparison  of  the  PODs  provided  a good  indication  of  the  order  of 
importance  of  the  influencing  parameters  on  the  NDE  sensitivity. 


6. 1.2. 3 Statistical  Fitting  Schemes 

The  parametric  relationships  between  the  POD  and  NDE  para- 
meters determined  by  the  adaptive  learning  techniques  and  the 
linear  regression  analysis  method  are  presented  in  the  following 
paragraphs.  Due  to  the  non-uniform  distribution  of  the  relia- 
bility data  caused  by  the  grouping  of  data  according  to  different 
NDE  parameters,  only  selected  comparisons  could  be  made  among 
the  large  number  of  combinations  of  NDE  parameters.  The  relation- 
ship established  by  the  adaptive  learning  techniques  is  in  the 
form  of  POD  (point  estimate)  plot  as  a function  of  crack  length. 
The  statistical  nature  of  the  fitting  program  was  such  that  the 
POD  was  allowed  to  reach  a maximum  of  1.1  in  order  to  arrive  at 
the  best  fit  for  the  data  points  at  the  small  crack  length  ranges. 
The  idea  of  a POD  in  excess  of  1.0  appears  to  be  absurd  at  first 
sight.  It  should  be  noted,  however,  that  the  majority  of  data 
points  were  in  the  lower  crack  length  ranges.  Consequently,  it 
will  be  more  appropriate  to  attempt  a better  fit  at  these  ranges 
at  the  expense  of  entertaining  a higher  than  1.0  POD.  The  pro- 
grams could  be  easily  changed  to  suppress  the  POD  to  a value  no 
larger  than  1.0,  but  it  would  be  accomplished  at  a price  of  arti- 
ficially shifting  down  the  POD  at  the  lower  crack  length  ranges. 

The  horizontal  axis  of  the  POD  curves  for  results  obtained 
by  the  adaptive  learning  techniques  as  well  as  the  linear  regres- 
sion analysis  is  divided  into  ten  equal  increments  of  0.064  in. 

It  should  be  noted,  however,  that  the  data  points  in  the  calcula- 
tion process  were  actually  selected  in  logarithmically  equal  in- 
crements of  crack  length.  The  equal  increments  in  logarithmic 
scale  were  used  instead  of  equal  increments  in  linear  scale  for 
the  purpose  of  achieving  a better  sampling  population  at  the  lower 
crack  length  ranges  where  the  POD  curves  generally  had  the  fastest 
change  in  slope.  In  order  to  present  the  POD  curves  at  a more 
conventional  way,  the  data  plots  were  changed  back  into  a familiar 
linear  crack  length  scale. 

The  computer  programs  developed  for  the  adaptive  learning 
techniques  generated  two  POD  curves  for  comparing  a pair  of 
values  for  each  NDE  parameter.  In  order  to  conserve  space  in 
the  presentation  of  these  comparisons,  POD  curves  for  several 
values  of  each  NDE  parameter  were  placed  in  the  same  figure  by 
transposing  the  curves.  The  surface  finish  and  specimen  thick- 
ness for  the  curves  are  identified  by  their  codes  (see  Table  4) 
as  a pair  of  numbers  in  a parenthesis.  For  example,  (1,2)  signi- 
fies that  the  surface  finish  of  the  specimen  was  in  the  range  of 
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1-32  rms  and  the  thickness  of  the  specimen  was  in  the  range  of 
201-300  mils.  Parametric  relationships  obtained  from  adaptive 
learning  technique  and  linear  regression  techniques  were  mostly 
for  ultrasonic  inspection  due  to  the  large  population  of 
inspection  data  for  this  NDE  technique.  Figure  4 shows  a 
comparison  of  POD  for  flat  plate  specimens  with  three  different 
surface  finishes.  The  POD  for  smoother  specimen  surface  finish 
was  higher  than  that  for  a rougher  surface.  Figure  5 presents 
a comparison  of  etched  and  unetched  specimen  surfaces  for  flat 
plate  and  integrally  stiffened  panel  specimens.  Specimens  with 
etched  surface  appear  to  have  a higher  POD  compared  to  specimens 
with  unetched  surfaces.  The  trend  applied  to  both  specimen 
geometries.  The  specimen  surface  finish  did  not  appear  to  affect 
the  POD  for  longitudinal  weld  without  crown  for  both  etched  and 
unetched  surfaces,  as  shown  in  Figure  6.  The  specimens  with 
etched  surfaces  had  a higher  POD  compared  to  those  with  unetched 
surfaces,  but  the  surface  finish  apparently  had  little  effect 
on  the  POD.  A similar  indifference  to  specimen  surface  finish 
was  evident  for  transverse  weld  specimens  with  crown  in  Figure  7. 

In  this  figure,  the  same  conclusion  could  be  reached  for  two 
different  specimen  thicknesses  and  for  etched  as  well  as  unetched 
surfaces.  However,  etching  the  surface  of  this  type  of  specimen 
tended  to  lower  the  POD  when  curves  C and  D are  compared  to  curves 
E and  F.  By  comparing  curves  A and  B to  curves  C and  D,  the  effect 
of  specimen  thickness  on  POD  for  the  transverse  weld  specimens  can 
be  observed. 

The  thesis  that  the  depth  to  length  ratio  (a/2c)  of  fatigue 
cracks  is  an  important  parameter  for  determining  the  POD  for  the 
ultrasonic  shear  wave  inspection  has  been  contended  by  many  NDE 
personnel.  However,  no  systematic  and  statistical  evidence  exists 
to  date  to  support  this  contention.  One  of  the  important  NDE 
parametric  relationships  established  in  this  program  was  that  a/ 2c 
was  indeed  a factor  in  POD  determination.  Figure  8 shows  that 
the  POD  curves  of  three  a/2c  values  for  flat  plate  specimens  were 
such  that  the  largest  depth  to  length  ratio  had  the  highest  PODs . 
Similar  trend  could  be  observed  in  Figure  9 for  integrally 
stiffened  panel  specimens.  It  will  be  noted  that  a comparison  of 
POD  for  cracks  with  the  same  length  can  be  obtained  from  Figures 
8 and  9 by  drawing  a vertical  line  from  the  desired  crack  length 
on  the  horizontal  axis.  The  vertical  line  intersects  the  three 
curves  at  three  different  points  corresponding  to  three  a/2c  values. 
Since  the  crack  length  2c  on  the  vertical  line  is  the  same,  the 
three  intersecting  points  will  represent  cracks  with  three 
different  depths.  The  ordinates  of  these  three  points  will 
represent  the  PODs  for  the  cracks  with  three  different  depths. 
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CRACK  LENGTH 

Figure  6 Effects  of  Specimen  History  L Surface  Finish  on  POD  by  Adaptive  Learning 
Technique  (Ultrasonic) 


0.32 

CRACK  LENGTH  (IN.) 

Figure  7 Effects  of  Specimen  History,  Surface  Finish  & Thickness  on  POD  by 
Adaptive  Learning  Technique  (Ultrasonic) 


CRACK  LENGTH  (IN.) 

Figure  8 Effects  of  a/2C  on  POD  by  Adaptive  Learning  Technique  (Ultrasonic 
Flat  Plate  Specimens) 


The  effects  of  specimen  history  and  surface  finish  on 
the  POD  of  integrally  stiffened  panel  and  transverse  weld 
specimens  for  the  penetrant  technique  by  using  the  linear 
regression  analysis  are  presented  in  Figures  10  and  11.  As 
expected,  higher  PODs  were  obtained  on  integrally  stiffened 
panel  specimens  after  the  surfaces  were  etched.  In  general, 
a smoother  surface  finish  produced  a higher  POD.  The  effect 
of  etching  the  specimen  surface  was  similar  in  transverse 
weld  specimens.  However,  the  effect  of  proof  loading  the 
specimens  to  90  percent  of  yielding  stress  did  not  appear  to 
increase  the  POD  to  a point  higher  than  that  for  specimens  with 
unetched  surfaces.  For  the  eddy  current  technique,  similar 
effects  of  the  specimen  surface  finish  were  observed  for  flat 
plate  specimens  from  the  linear  regression  analysis.  Figure 
12  shows  the  difference  in  POD  at  different  crack  length  ranges 
for  specimens  with  etched  and  unetched  surfaces. 

An  interesting  comparison  of  POD  curves  of  flat  plate 
specimens  with  unetched  surfaces  for  the  ultrasonic,  penetrant, 
and  eddy  current  techniques  from  the  linear  regression  analysis 
is  shown  in  Figure  13.  An  average  difference  in  POD  of  0.2  was 
seen  to  exist  between  the  eddy  current/ultrasonic  and  ultrasonic/ 
penetrant  techniques.  The  corresponding  POD  curves  for  flat 
plate  specimens  with  etched  surfaces  are  shown  in  Figure  14. 

The  POD  curve  for  the  penetrant  technique  is  seen  to  approach 
that  for  the  ultrasonic  technique  after  the  specimen  surfaces 
were  etched  as  shown  by  the  solid  line.  However,  etching  the 
surface  appears  to  have  a reversed  effect  for  the  eddy  current 
technique.  The  PODs  at  the  lower  crack  length  were  actually 
lower  after  etching.  This  result  is  in  good  agreement  with  that 
obtained  by  the  point  estimate  scheme. 

6. 1.2. 4 Summary  of  Parametric  Relationship  Study 

Summaries  of  the  parametric  relationship  for  the  ultrasonic, 
penetrant,  and  eddy  current  techniques  are  tabulated  in  Tables 
5,  6 and  7,  respectively.  These  summaries  are  comprised  of 
matrices  of  four  specimen  geometries  and  nine  NDE  parameters.  A 
dash  mark  in  the  matrix  indicates  insufficient  or  no  data  avail- 
able. YES  indicates  a definite  relationship  exists  while  NO 
signifies  negligible  or  no  effect  on  POD.  Although  these  para- 
metric relationships  were  often  established  by  one  or  more 
analytical  or  fitting  techniques,  the  degrees  of  confidence  based 
on  the  number  of  data  points  in  most  cases  were  such  that  a quali- 
tative evaluation  was  judged  to  be  more  appropriate. 
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CRACK  LENGTH  (IN.) 

Figure  11  Effects  of  Specimen  History  on  POD  by  Linear  Regression  Technique  (Penetrant) 
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Effects  of  Specimen  History  on  POD  by  Linear  Regression  Technique 
(Eddy  Current) 


TABLE 


CONCLUSION  - PARAMETRIC  RELATIONSHIP,  PENETRANT 


CONCLUSION  - PARAMETRIC  RELATIONSHIP,  EDDY  CURRENT 


The  summaries  of  parametric  relationships  indicated  that  the 
inspection  environment  had  no  effect  on  the  POD  for  all  three 
inspection  techniques.  The  inspection  environments  studied  in 
the  comparison  were  laboratory  and  production  environments  where 
the  capabilities  of  operators  did  not  differ  significantly.  Only 
in  isolated  cases  the  performance  of  an  operator  would  fall  below 
average  for  a certain  reason.  A final  observation  was  that  the 
defect  types  of  fatigue  crack  and  EDM  compressed  notches  did  not 
have  significant  effect  on  the  POD  in  the  case  of  ultrasonic 
inspection.  For  penetrant  and  eddy  current  techniques,  the  dif- 
ference in  defect  type  did  produce  a difference  in  POD  as  expected. 


6.2  TRANSLATION  MODEL 


Based  on  the  parametric  relationships  established  between 
the  NDE  parameters  and  POD,  translation  models  were  developed 
for  the  ultrasonic,  penetrant,  and  eddy  current  techniques. 

The  models  could  be  used  to  translate  the  POD  obtained  on  flat 
plate  specimens  to  specimens  with  other  geometries  such  as 
integrally  stiffened  panel,  longitudinal  weld,  transverse  weld, 
and  tandem  T.  The  models  developed  by  adaptive  learning  tech- 
niques are  expressed  in  the  form  of  POD  curves  for  the  flat 
plate  geometry  and  the  geometry  to  be  translated.  Models  de- 
veloped by  the  linear  regression  analysis  are  best  expressed  in 
the  form  of  difference  tables.  These  tables  are  presented  in 
Appendix  E for  reference  purposes.  For  convenience  in  compari- 
son, the  linear  regression  models  are  also  presented  in  graphical 
form  and  discussed  in  conjunction  with  results  obtained  from 
adaptive  learning  techniques.  These  results  will  be  presented 
according  to  NDE  techniques.  In  addition  to  the  statistical 
fitting  schemes,  a point  estimate  transfer  function  approach  was 
also  used  to  assess  the  difference  in  POD  for  different  specimen 
geometries.  Results  of  this  comparison  are  rather  interesting 
and  will  be  presented  in  a separate  paragraph. 


6.2.1  Ultrasonic  Inspection 

The  translation  models  developed  by  the  adaptive  learning 
and  linear  regression  techniques  for  ultrasonic  inspection  from 
flat  plate  to  integrally  stiffened  panel  specimens  are  shown  in 
Figures  15  and  16,  respectively.  Curves  A and  C in  Figure  15 
were  POD  curves  for  flat  plate  specimens  while  curves  B and  D 
were  for  integrally  stiffened  panel  specimens.  A significant 
difference  is  seen  to  exist  between  the  two  POD  curves.  Curves 
A and  B were  obtained  by  using  a weighting  factor  to  account  for 
the  decreasing  significance  in  curve  fitting  for  crack  ranges 
where  data  points  were  scarce.  Curves  C and  D were  equivalent 
curves  obtained  by  not  using  the  weighting  factor.  It  is  seen 
that  the  difference  with  and  without  the  factor  was  not  significant. 
All  the  curves  obtained  by  using  the  adaptive  learning  technique 
incorporated  the  weighting  factors  which  were  merely  concerned 
with  the  curve  fitting  process  and  had  no  impact  on  the  calcu- 
lation of  probability  of  detection.  From  the  linear  regression 
model  shown  in  Figure  16,  it  can  be  seen  that  the  basic  model 
agreed  well  with  that  shown  in  Figure  15.  The  difference  in  POD 
for  two  moHo  1 s diss^rs^d  scir.sv^hsti. 
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Figure  1^  Translation  Model  by  Linear  Regression  Tdchnique  - Flat  Plate/ 
Integrally  Stiffened  Panel  (Ultrasonic) 


Figures  17  and  18  show  the  translation  models  developed 
from  the  adaptive  learning  and  linear  regression  techniques, 
respectively,  for  the  translation  from  flat  plate  to  longitudinal 
weld  specimens.  Although  a large  difference  in  POD  was  observed 
in  Figure  17,  an  inconsistent  difference  for  various  crack 
lengths  existed  in  Figure  18.  A similar  situation  was  noted 
for  the  translation  from  flat  plate  to  transverse  weld  specimens 
as  shown  in  Figures  19  and  20  for  adaptive  learning  and  linear 
regression  techniques,  respectively.  Figure  21  shows  the 
adaptive  learning  translation  model  from  flat  plate  tc  tandem  T 
specimens.  A small  and  almost  negligible  difference  exists  in 
the  POD  curves  for  these  specimen  geometries. 

6.2.2  Penetrant  Inspection 

The  adaptive  learning  and  linear  regression  translation 
models  for  the  penetrant  technique  translating  flat  plate  to 
integrally  stiffened  panel  specimens  are  Dresented  in  Figures 
22  and  23,  respectively.  A sizeable  difference  in  POD  curves 
was  observed  in  both  figures  for  the  two  geometries.  In  Figures 
24  and  25,  essentially  no  systematic  difference  in  POD  curves 
was  observed  for  the  geometries  of  flat  plate  to  longitudinal 
weld  specimens.  The  same  is  true  for  the  translation  from  flat 
plate  to  transverse  weld  specimens  as  shown  in  Figures  26  and  27 
for  the  adaptive  learning  and  linear  regression  models,  respec- 
tively. Basically,  the  models  for  the  penetrant  technique 
followed  closely  the  pattern  set  by  the  ultrasonic  technique. 

The  quantitative  differential,  however,  was  slightly  decreased. 

6.2.3  Eddy  Current  Inspection 

The  adaptive  learning  and  linear  regression  translation 
models  for  the  eddy  current  inspection  translating  flat  plate 
to  integrally  stiffened  and  flat  plate  specimens  to  longitudinal 
weld  followed  the  same  trend  for  the  ultrasonic  and  penetrant 
inspection.  These  models  are  shown  in  Figures  28  to  31.  A large 
difference  in  POD  curve  was  observed  in  the  adaptive  learning 
model  translating  flat  plate  to  transverse  weld  specimens  as 
shown  in  Figure  32.  The  corresponding  linear  regression  model 
presented  in  Figure  33  showed  inconsistent  differences  at  different 
crack  ranges.  An  interesting  comparison  of  POD  curves  for  flat 
plate  and  bolt  hole  geometries  from  adaptive  learning  technique 
is  presented  in  Figure  34.  The  comparison  is  termed  translation 
model  only  in  the  sense  that  both  specimen  geometries  contained 
fatigue  cracks.  The  insnpof'on  environment  for  the  bolt  hole 
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Figure  17  Translation  Model  by  Adaptive  Learning  Technique  - Flat  Plate/ 
Long.  Flush  Weld  (Ultrasonic) 


Model  by  Linear  Regression  Technique  - Flat  Plate/Long.  Flus 
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Figure  IP  Translation  Model  by  Adaptive  Learning  Technique  - Flat  Plate/Transverse 
Weld  w/Crown  (Ultrasonic) 


CRACK  LENGTH 

Figure  20  Translation  Model  by  Linear  Regression  Technique  - Flat  Plate/Transverse 
Weld  w/Crown  (Ultrasonic) 
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Figure  24  Translation  Model  by  Adaptive  Learning  Technique  - Flat  Plate/Long 
Flush  Weld  (Penetrant) 
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ranslation  Model  Adaptive  Learning  Technique  - Flat  Plate/Transverse  Weld 
/Crown  (Penetrant) 
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Figure  27  Transla^  iodel  by  Linear  Regression  Technique  - Flat.  Plate/ 
Transverse  vJeld  w/Crown  (Penetrant) 


CRACK  LENGTH  (IN.) 

Figure  28  Translational  Model  by  Adaptive  Learning  Technique  - Flat  Plate/ 
Integrally  Stiffened  Panel  (Eddy  Current) 
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Translation  Model  by  Adaptive  Learning  Technique  - Flat  Plate/Long 


CRACK  LENGTH  (IN.) 

Translation  Modei  by  Adaptive  Learning  Technique  - Flat  Plate/Transverse 
Weld  w/Crown  (Eddy  Current) 
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Figure  33  Translation  Model  by  Linear  Regression  Technique  - Flat  Plate/ 
Transverse  Weld  w/Crown  (Eddy  Current) 
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specimens  was  a field  and  depot  type  inspection  while  the  flat 
plate  geometry  inspection  was  conducted  in  a laboratory  environ- 
ment with  an  automatic  scanning  device.  The  bolt  hole  POD 
curve  is  the  only  one  available  for  model  development  in  the 
data  bank. 


6.2.4  Transfer  Function 

The  NDE  methods  sensitivity  for  each  specimen  geometry  was 
studied  for  the  ultrasonic,  penetrant,  and  eddy  current  inspec- 
tions by  pooling  all  the  data  pertaining  to  that  particular 
geometry  without  separating  the  other  NDE  parameters.  The 
PODs  were  calculated  by  using  the  point  estimate  method.  Figures 
35  through  43  show  the  POD  curves  for  the  geometries  of  flat 
plate,  integrally  stiffened  panel, longitudinal  flush  weld,  trans- 
verse weld  with  crown,  longitudinal  weld  with  crown,  riveted  plate 
to  integrally  stiffened  panel,  weld  panels  with  LOP,  bolt  holes, 
and  tandem  T , respectively.  Each  of  these  figures  contains  data 
points  for  three  inspection  techniques.  The  total  number  of 
data  points  (N)  for  each  inspection  technique  is  indicated  in 
parenthesis.  For  the  sake  of  clarity,  only  a curve  joining  the 
ultrasonic  data  points  was  drawn  in  each  figure. 

The  flat  plate  specimens  contained  the  largest  number  of 
data  points.  They  represent  the  reliability  data  generated  by 
several  companies  using  the  same  specimens.  These  data  are 
considered  to  be  the  most  complete  and  well-balanced  data  sets 
in  the  reliability  data  bank.  The  well-behaved  POD  curves  for 
this  specimen  geometry  reflect  the  large  population  of  data  point 
in  Figure  35.  The  second  most  populous  curve  is  the  POD  curve 
for  bolt  holes  for  the  eddy  current  inspection  (N=1896) . The 
curve  is  also  well-behaved  although  the  POD  levels  for  the  curve 
were  rather  low. 

A transfer  function  to  relate  the  ultrasonic  data  from  flat 
plates  to  specimens  with  each  of  the  other  geometries  can  be 
obtained  from  the  data  in  Figures  35  to  43.  It  is  defined  by  the 
following  relationship: 


Transfer  Function 


POD  (Other  Geometries) 
POD  (Flat  Plate) 


The  transfer  function  for  translating  POD  curves  from  flat  plate 
to  integrally  stiffened  panel  specimens  was  calculated  and  plotted 
in  Figure  44.  Similar  curves  could  be  generated  for  the  other 
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Figure  36  NDE  Methods  Sensitivity  for  Integrally  Stiffened  Panels 
bv  Point  Estimate  Method 
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'E  Methods  Sensitivity  for  Transverse  Weld  w/Crown  by  Point  Estimate  Method 
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Methods  Sensitivity  for  Bolt  Holes  by  Point  Estimate  Method 


CRACK  LENGTH  (IN.) 

Methods  Sensitivity  for  Tandem  T by  Point  Estimate  Method 


geometries.  The  main  purpose  of  the  composite  POD  curves, 
however,  is  to  serve  as  a check  for  the  adaptive  learning  and 
linear  regression  translation  models  discussed  earlier.  The 
comparison  of  the  point  estimate  results  with  the  adaptive 
learning  and  linear  regression  results  can  be  easily  done  by 
transposing  POD  curves  in  Figures  36  through  43  to  the  flat 
plate  curve  in  Figure  35.  The  comparison  substantiated  the 
results  obtained  by  using  the  adaptive  learning  and  linear 
regression. 


6.2.5  Summary  of  Translation  Model  Development 

A summary  of  the  translation  model  development  results 
using  the  adaptive  learning,  linear  regression  and  point 
estimate  methods  is  presented  in  Table  8.  The  PODs  for  the 
flat  plate  specimens  were  used  as  a common  reference  basis. 

PODs  for  specimens  with  other  geometries  were  compared  with 
the  reference  basis.  The  magnitude  of  the  differences  was 
expressed  in  a semi-quantitative  term  for  comparison  purpose. 

The  results  of  the  comparison  indicated  that  a good  agreement 
existed  for  the  translation  models  from  flat  plate  to  integrally 
stiffened  panel  specimens  for  the  three  analytical  techniques. 

Except  for  a degree  of  difference  in  the  case  of  the  ultrasonic 
technique  using  the  adaptive  learning  technique,  the  models  for 
the  three  techniques  were  in  good  agreement. 

The  successful  development  of  a model  to  translate  NDE 
sensitivity  from  flat  plate  to  integrally  stiffened  panel  specimens 
for  the  ultrasonic,  penetrant,  and  eddy  current  inspections  is 
rather  useful.  The  integrally  stiffened  panel  specimens  represented 
specimens  with  fillet  areas  where  the  fatigue  cracks  were  located. 

It  has  been  estimated  that  fillet  areas  are  the  second  most 
fatigue  critical  areas  in  advanced  fighter  and  bomber  components. 

The  translation  model  for  this  geometry  can  be  useful  in  the 
interpretation  of  NDE  capability  determination  in  demonstration 
programs . 
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6.3  DATA  DEFICIENCIES 
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The  reliability  data  in  the  computer  data  bank  contains 
primarily  well  documented  data  on  aluminum  flat  plates,  inte- 
grally stiffened  panels,  and  weld  joints.  From  the  available 
data,  realistic  translation  models  were  developed  to  translate 
NDE  inspection  results  obtained  from  flat  plate  specimens  to 
specimens  with  fillet  areas.  NDE  data  on  bolt  hole  specimens 
are  much  needed.  The  fatigue  fracture  and  durability  critical 
part  geometries  for  F-16  and  B-l  are  summarized  in  Table  9. 

It  can  be  seen  that  fillets  and  curved  surfaces  are  the  second 
most  numerous  geometries  accounting  for  22.2  and  13.2  percent 
of  the  total  critical  part  geometries  for  F-16  and  B-l,  re- 
spectively. The  fastener  holes  and  cutouts  for  F-16  and  B-l 
account  for  72.2  and  78  percent  of  the  total  critical  part 
geometries.  The  importance  of  a translation  model  for  fastener 
hole  inspections  is  evident  from  this  analysis.  The  methodology 
of  the  translation  model  from  flat  plate  specimens  to  integrally 
stiffened  panels  can  be  applied  to  cover  the  flat  plate/fastener 
hole  geometries.  Presently,  only  a limited  amount  of  fastener 
hole  inspection  results  have  been  entered  into  the  computer  data 
bank. 

A search  for  possible  remedies  to  overcome  the  fastener 
hole  inspection  data  deficiency  revealed  that  a potentially 
useful  data  source  exists.  Under  a previous  Air  Force  contract 
for  the  F-lll  tear  down  inspection  at  General  Dynamics,  74  compo- 
nents of  F-lll  fatigue  tested  were  inspected  by  eddy  current, 
magnetic  particle,  and  penetrant  techniques (8 >9) t Approximately 
16,000  fastener  holes  were  inspected  by  using  these  three  tech- 
niques. The  results  and  inspection  procedures  were  well  docu- 
mented. However,  the  approximately  400  positive  indications  of 
fatigue  cracks  in  these  components  had  not  been  verified.  If 
a portion  of  these  positive  indications  can  be  selectively 
verified  by  destructive  testing,  the  inspection  results  from 
the  tear-down  inspections  can  be  a large  block  of  useful  relia- 
bility data. 

Few  NDE  reliability  data  suitable  for  translation  model 
development  exists  in  the  data  bank  for  steel  components  and  for 
the  inspection  techniques  of  magnetic  particle  and  X-ray.  The 
numbers  of  x-ray  inspections  for  the  aluminum  flat  plate  specimens 
containing  fatigue  cracks  in  the  data  bank  were  found  to  be  insuf- 
ficient for  statistical  calculation.  The  PODs  for  the  inspections 
were  quite  low  which  is  understandable  since  it  is  generally 
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TABLE  9 

ESTIMATE  OF  FATIGUE  FRACTURE 
AND  DURABILITY  CRITICAL  PARTS  FOR  F-16  AND  B-l 


Critical  Part 
Geometry 

Fastener  Holes 

Cutouts 

Fillets 

Weld 

Curved  Surfaces 
Internal  Threads 
Surface  Edges 


F-16 

3 

10 

4 


B-l 

65 

6 

4 
2 
8 
1 

5 


TOTAL 


18 
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recognized  that  fatigue  cracks  are  difficult  to  detect  by  using 
X-ray  technique.  In  aircraft  inspections.  X-ray  is  more  often 
used  for  the  inspection  of  forgings  and  welds.  The  steel 
specimens  used  in  the  AFML  program  on  Practical  Sensitivity 
Limits  could  be  used  to  generate  more  data  on  X-ray  and 

magnetic  particle  inspection  methods.  In  addition,  the  numbers 
of  inspections  in  the  program  (Reference  7)  for  cylindrically 
shaped  specimens  containing  EDM  compressed  notch  flaws  were  not 
sufficient  to  allow  a statistically  valid  analysis.  It  is 
recommended  that  specimens  used  in  that  program  be  selectively 
used  in  a future  validation  program  to  augment  the  number  of 
inspections . 

From  the  standpoint  of  NDE  parameters  in  the  parametric 
studies,  certain  deficiencies  were  discovered  in  the  existing 
reliability  data.  Inspection  data  on  specimens  with  surface 
finishes  in  the  range  from  65  to  128  RMS  were  inadequate.  Un- 
fortunately, this  range  happened  to  be  the  finish  of  the  as- 
machined  surfaces  of  most  aircraft  components.  For  some  NDE 
techniques  such  as  eddy  current  and  liquid  penetrant,  it  was 
shown  that  the  surface  finish  could  affect  the  inspection  sensi- 
tivity. Thus,  it  is  important  to  supplement  the  data  inadequacy 
in  this  range  of  commonly  occurring  surface  finishes  in  order  to 
extend  the  validity  of  the  translation  models.  For  the  NDE 
parameter  of  specimen  thickness,  the  majority  of  the  data  were 
obtained  on  specimens  with  thicknesses  below  0.3  in.  In  cases 
where  the  specimen  thickness  does  affect  the  inspection  sensi- 
tivity, especially  for  the  NDE  techniques  of  X-ray,  magnetic 
particle,  and  ultrasonic,  it  is  desirable  to  obtain  more  data  on 
specimens  thicker  than  0.3  in.  The  full  extent  of  how  the 
specimen  thickness  affects  the  inspection  results  will  not  be 
known  until  this  data  deficiency  is  overcome. 

A serious  deficiency  in  the  reliability  data  bank  is 
connected  with  the  ultrasonic  inspection.  It  is  the  second  most 
used  technique  in  production  inspection.  The  majority  of  the  data 
on  ultrasonic  inspection  in  the  data  bank  related  to  shear  wave 
inspection.  Data  on  the  commonly  used  modes  in  ultrasonic 
technique,  immersion  mode  by  compression  waves  and  contact  mode  by 
shear  wave,  were  grossly  inadequate.  In  a demonstration  program 
designed  to  determine  the  NDE  capability  of  a production  facility, 
it  is  important  to  assess  the  inspection  efficiency  of  the  equip- 
ment and  the  personnel  in  the  area  of  ultrasonic  immersion 
inspection.  It  will  be  noted  that  the  ultrasonic  technique  in  the 
immersion  mode  of  operation  is  still  one  of  the  most  widely  used 
techniques  in  the  field  of  NDE  when  the  entire  spectrum  of  materials 
and  finished  products  in  the  aircraft  industry  is  considered.  The 
deficiency  in  the  reliability  data  in  this  area  must  be  remedied. 
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SECTION 


VII 


OPTIMUM  DEMONSTRATION 
PROGRAM  DESIGN 


7.1  INTRODUCTION 

An  optimum  demonstration  program  is  required  in  order 
to  establish  a baseline  procedural  method  to  ensure 
that  the  requirements  regarding  the  capabilities  of  the 
nondestructive  inspection  process  to  meet  the  MIL  Specification 
requirements  be  firmly  established.  In  most,  if  not  all, 
of  the  earlier  programs  designed  to  evaluate  the  probability 
of  detection  of  a particular  inspection  process,  there  were 
overwhelming  external  influences  that  dominated  the  various 
decisions  made  during  the  conduct  of  the  demonstration  program. 
For  example,  the  Rockwell  B-l  demonstration  program  was  in 
fact  a go-no-go  program  in  which  the  requirement  was  to  prove 
that  the  minimum  flaw  size  used  for  the  fracture  design  analysis 
of  the  specified  fracture  critical  components  could  be  detected 
to  the  predetermined  degree  of  confidence  and  probability  of 
detection.  On  the  other  hand,  subsequent  subcontractor  demon- 
stration programs  were  limited  as  to  the  ranges  of  flaws, 
numbers  of  observations  within  each  flaw  size  range,  type  of 
flaws  used,  significance  of  the  reuse  of  specimens  and  substi- 
tuteability  of  inspection  parameters,  etc.  Hence,  one  must  be 
extremely  careful  in  stating  the  initial  objective  and  statis- 
tical hypothesis  that  is  to  be  tested  and  to  carefully  evaluate 
the  demonstration  program  to  ensure  that  it  does  indeed  test 
this  hypothesis.  For  the  optimum  demonstration  program  discussed 
here,  the  hypothesis  to  be  tested  may  be  stated  as  follows: 

"Can  a flaw  size  2c  (inches)  be  detected  by  the 
nondestructive  process  under  investigation  to  a minimum 
level  of  90  percent  probability  of  detection  at  a 95  percent 
confidence  level." 
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This  hypothesis  immediately  eliminates  a great  deal  of  extra 
testing,  since  no  judgments  are  to  be  made  regarding  the  POD 
of  flaw  sizes  in  immediately  adjacent  ranges.  Implied  in  the 
process  is  the  fact  that  flaws  whose  sizes  are  larger  than 
the  required  2c  will  be  detected  to  at  least  that  degree 
(90/95)  and  probably  higher.  The  required  POD  is  a lower 
bound  value,  and  the  true  value  of  detection  probability  is 
assumed  to  be  higher  (at  the  95%  confidence  level).  No 
estimate  is  to  be  made  of  the  true  value  of  the  detection 
probability,  i.e.,  the  statement  impresses  the  fact  that  the 
lower  bound  detection  probability  level  is  90%,  but  does  not 
make  any  statements  as  to  the  value  of  the  upper  bound.  To 
make  a statement  that  the  probability  of  detection  is  between 
90  and  95%  probability  of  detection  at  a 95%  confidence  level 
would  take  a great  deal  more  specimens  than  that  required  by 
this  optimum  demonstration  program. 

It  should  be  noted  that  this  hypothesis  differs  from  the 
recommended  practice  of  ASNT  "Demonstration  of  Nondestructive 
Evaluation  Reliability  on  Aircraft  Production  Parts"  Draft  2.'^®' 
The  purpose  of  the  ASNT  recommended  practice  is  to  develop 
repeatable  data  for  fracture  mechanics  applications.  It  does  not 
include  apriori  requirements  as  to  the  required  degree  of  detection 
and  confidence,  but  allows  for  several  combinations  of  probability 
of  detection  and  confidence  levels. 


The  choice  of  the  flaw  size  2c  to  be  demonstrated  in  the 

* 

optimum  demonstration  program  can  be  evaluated  by  use  of  the 
transfer  function  model  approach.  The  transfer  function  gives 
the  ratio  of  the  POD  for  the  complex  specimen  to  the  POD  for  the 
simple  configuration,  as  a function  of  flaw  size.  Thus,  if  the 
requirement  would  be  to  detect  a 0.10  inch  defect  in  a complex 
structure,  this  would  be  the  equivalent  of  a (POD  /POD,.  x 0.010) 
size  defect  in  a flat  plate.  If  the  transfer  function  were  0.80 
at  a 0.10  inch  defects  for  ultrasonics,  this  would  mean  that  a 2c 
value  of  0.080  inches  in  length  on  the  flat  plate  would  demonstrate 
the  equivalent  POD  for  the  complex  specimen.  Thus  the  target  flaw 
sizes  would  be  0.08  inches  in  length  in  the  flat  plate.  It  should 
also  be  determined  that  the  POD  curve  for  the  NDI  method  of 
inspection  does  not  peak  in  that  crack  range. 


7.2  SPECIMEN  CONFIGURATION 


In  the  choice  of  a suitable  specimen  geometry,  two 
conflicting  factors  must  be  considered:  (1)  simplicity  of 
specimen  configuration  and  (2)  similarity  of  the  specimen 
to  the  fracture  critical  component  the  demonstration  program 
is  to  represent. 

The  demonstration  specimens  should  simulate  the 
fracture  critical  components  with  respect  to  alloy,  heat 
treat  condition,  metallurgical  microstructure,  primary  and 
secondary  processing  variables.  It  should  furthermore  be 
representative  of  the  surface  finish,  texture,  surface 
condition  (i.e.  stress  state  of  surface)  and  manufacturing 
parameters  of  the  fracture  critical  component.  It  is  imme- 
diately obvious  that  the  validity  of  a demonstration  program 
to  determine  the  POD  of  penetrants  obtained  on  castings  to 
be  used  on  forgings  is  doubtful.  This  would  be  true  even  if 
all  other  metallurgical  and  processing  variables  were  identical. 

Within  these  limitations  as  to  the  metallurgical 
similarity  of  the  specimen  type,  the  geometry  of  the  specimen 
should  be  kept  as  simple  as  possible  so  that  a controlled 
flaw  can  be  produced. 

The  validity  of  the  hypothesis  depends  upon  the  degree 
to  which  the  specimen-defect  configuration  can  be  controlled 
and  predetermined.  For  this  reason,  flat  plate  specimens  would 
be  suitable  for  most  demonstration  purposes.  These  plates  may 
contain  welds  or  bolt  holes  if  the  demonstration  program  is 
concerned  with  this  type  of  fracture  critical  component. 

The  effects  of  specimen  geometry  of  the  POD  for  four 
NDE  techniques  revealed  that  the  detection  probabilities  are 
higher  in  the  simpler  geometry  than  with  the  integrally  stiffened 
panels.  Both  specimens  contained  fatigue  cracks  and  were  in- 
spected by  penetrant,  eddy  current  and  ultrasonics.  However, 
the  differences  in  POD  were  only  found  to  be  significant  for 
the  smallest  crack  length  in  the  case  of  the  ultrasonic  technique, 
(c  . f . Figure  D- 11) . 
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Specimen  Size: 


The  size  of  the  plate  should  be  consistent  with  the 
test  equipment  available  to  produce  suitable  defects  within 
an  optimum  time  frame.  Specimen  configuration  should  be  such 
that  a minimum  of  4"  by  8"  of  finished  specimen  size  is 
obtained.  To  produce  a finished  specimen  size  with  the  defect 
randomly  located  within  the  specimen,  an  initial  specimen  size 
and  configuration  consistent  with  the  method  of  producing  the 
flaw  should  be  developed.  For  example,  if  the  defects  are 
to  be  fatigue  cracks,  suitable  three  or  four  point  bending 
equipment  will  determine  the  initial  size  of  the  specimen,  with 
the  added  dimensional  requirements  dominated  by  the  choice  of 
loading  fixture. 

For  most  demonstration  programs,  the  location  of  the 
defect  within  the  4x8  area  is  not  critical,  except  that  it 
should  not  be  too  close  to  any  of  the  edges.  Thus,  location 
of  the  defect  within  the  central  3x6  inches  would  be  preferable. 
However,  if  edge  cracks  would  be  expected  to  be  the  dominant 
failure  mode,  i.e.  turbine  blades,  the  specimen  defect  location 
should  consider  this  factor  in  selection  of  the  locations. 

Flaw  Type: 

Although  prudence  would  recommend  that  the  flaw  type  be 
consistent  with  the  defect  type  expected  to  be  found  in  the 
preproduction  inspection  program,  experience  has  shown  that  small 
tightly  closed  cracks  are  the  most  difficult  to  detect  (except 
perhaps  for  forging  laps  in  diffusion  bonds),  and  most  closely 
represent  the  type  of  defect  for  which  the  fracture  mechanics 
analysis  is  valid.  Thus  tight  fatigue  type  cracks,  while  rare ly 
encountered  in  preproduction  components  would  impose  the  most 
severe,  and  hence  the  most  conservative,  estimate  of  the  POD  of 
the  particular  NDT  procedure. 

Fatigue  cracks  are  recommended  as  the  most  convenient 
defect  type  because  of  the  ease  with  which  their  size  may  be 
controlled,  and  the  large  body  of  metallurgical  and  fracture 
mechanics  literature  available  to  fully  evaluate  their  charac- 
teristics. Fatigue  cracks  should  be  initiated  in  the  pre- 
selected (randomized)  location  within  the  specimen.  The 
location  of  the  initiation  site  can  be  induced  by  a suitable 
choice  of  stress  riser  on  the  surface  of  the  specimen. 


Suitable  stress  risers  are:  electrodischarge  machining  (EDM), 
weld  solidification  spots,  microhardness  indentations,  weld 
beads,  and  conventional  machined  slots.  The  size,  shape  and 
depth  of  the  initiation  procedure  will  influence  the  shape 
and  extent  of  the  fatigue  crack  in  a manner  that  must  be  deter- 
mined for  each  testing  machine.  It  is  important  that  the 
starter  site  be  as  shallow  as  possible  so  that  it  may  be 
completely  removed  by  subsequent  surface  machining  procedures . 
In  this  respect,  it  should  be  recognized  that  EDM  and 
welding  initiation  sites  would  be  marginally  suitable  for 
magnetic  particle  inspection  procedures  due  to  the  magnetic 
field  perturbation  introduced  by  the  solidification  spot  which 
extends  considerably  beyond  the  geometric  initiation  size. 

In  the  case  of  ultrasonic  inspection  techniques, 
elox  slots  that  had  been  closed  by  forging  showed  approximately 
the  same  POD  as  did  defects  produced  by  fatigue.  However, 
elox  slots  are  not  identical  in  nature  to  fatigue  cracks  when 
examined  by  eddy  current  and  penetrant  processes.  Hence,  in 
certain  instances  the  compressed  EDM  slot  may  be  substituted 
for  the  fatigue  crack,  only  if  unambiguous  information  is 
available  regarding  the  degree  of  severity  of  the  different 
types  of  defects. 


Defect  Characterization 


In  all  instances  the  defect  used  in  the  demonstration 
program  should  be  characterized  as  fully  as  possible,  and  a 
record  of  these  characteristics  included  in  the  documentation 
of  the  demonstration  program.  These  should  include  the  stress 
ratios  used  to  fatigue  the  specimens,  the  presence  of  or 
absence  of  subsequent  corrosive  conditions,  subsequent  tensile 
or  compressive  loadings,  number  of  times  the  specimen  was  reused, 
and  the  history  of  all  prior  NOT  procedures  that  have  been 
examined  using  this  particular  specimen.  It  can  be  seen  that 
if  a specimen  had  been  examined  first  by  contact  shear  wave 
ultrasonics  using  oil  at  the  contact  medium,  and  then  by 
penetrant  inspection,  the  POD  for  the  penetrant  procedure 
may  be  decreased  due  to  the  presence  of  oil  from  the  initial 
inspection. 
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Fatigue  Crack  Growth 


The  conditions  for  the  fatigue  crack  growth  should  also 
be  standardized.  It  is  recommended  that  tension-tension  fatigue 
with  R values  of  -0.3  or  greater  be  used  in  producing  the  fatigue 
c racks . This  has  been  found  to  produce  cracks  whose  crack  opening 
displacements  are  smaller,  i.e.,  more  tightly  closed  than  cracks 
produced  by  reversed  bending  or  zero-max  fatigue. 

Defect  Orientation 


Defect  orientation  should  be  consistent  with  the 
procedure  used  to  produce  the  defects.  In  the  case  of  fatigue 
cracks,  they  will  most  often  have  to  be  oriented  perpendicular 
to  the  longitudinal  axis  of  the  specimen.  However,  in  most 
fracture  critical  components,  the  anticipated  crack  orientation 
would  be  known,  and  this  factor  is  not  considered  to  have  a 
significant  influence  on  the  POD  for  the  inspection  procedure. 
The  crack  orientation  would  have  a major  effect  in  the  case  of 
the  magnetic  particle  inspection  technique,  where  combinations 
of  crack  orientation  and  inspection  coil  configurations  could 
result  in  abnormally  low  POD  results. 

Defect  Geometry 

Defect  geometry  should  be  given  serious  consideration 
in  choosing  the  type  of  loading  procedure  used  to  grow  the 
fatigue  cracks.  The  defect  depth  to  surface  length  ratio 
a/2c  is  determined  by  the  shape  of  the  starter  notch,  the 
loading  fixture,  and  the  defect  depth  to  specimen  thickness 
ratio.  For  eddy  current  procedures  the  depth  would  be  an 
important  factor  controlling  the  POD,  but  for  ultrasonics 
there  is  an  effect  of  the  a/2c  ratio  on  the  POD,  c.f.  Figures 
8 and  9.  As  would  be  expected,  given  a value  of  surface  crack 
length,  the  POD  increases  as  the  depth  of  the  defect  increases, 
i.e.  the  reflected  defect  area  increases.  Thus,  the  POD  may 
be  adversely  affected  in  the  optimum  demonstration  program  if 
too  much  of  the  surface  is  machined  off  to  produce  the  required 
2c  length  leaving  a defect  of  the  desired  length,  but  with  a 
shallow  depth.  The  expected  final  a/2c  ratio  for  the  specimens 
should  be  held  as  close  to  constant  as  possible.  This  may  be 


accomplished  by  considering  the  a/2c  ratio  of  the  defect 
as  fatigued,  and  calculating  the  a/2c  ratio  of  the  defect 
after  a surface  depth  containing  the  starter  notch  has 
been  machined  off.  If  the  resulting  a/2c  value  is  too 
small,  i.e.,  less  than  0.2,  consideration  should  be  given 
to  further  fatigue  cycling. 

Surface  Condition 


Surface  finish  has  been  demonstrated  to  have  an  effect 
on  the  POD  for  ultrasonics,  eddy  current  and  penetrants  (Tables 
5 , 6 , 7 ) . It  is  therefore  recommended  that  the  surface  finish 
be  consistent  with  the  finish  to  be  used  on  the  fracture  critical 
components,  typically  125  RMS  or  better.  However,  consideration 
should  also  be  given  to  reproduce  the  type  of  machining  pro- 
cedures actually  used  on  the  fracture  critical  components.  For 
example,  the  depth  of  the  final  cut  may  influence  the  residual 
stress  state  on  the  component  and  hence  the  POD.  The  direction 
of  the  cutting  marks  relative  to  the  orientation  of  the  defect 
may  also  be  significant.  The. surface  finish  determines  the 
amount  of  surface  material  to  be  removed  by  etching  prior 
to  inspection  by  penetrant  procedures. 

An  equal  number  of  control  specimens  shall  be  prepared 
in  a manner  identical  to  that  used  in  preparing  the  samples 
containing  defects.  These  shall  be  numbered  in  a manner  as  to 
intersperse  them  within  the  specimens  containing  defects.  These 
should  be  verified  as  being  "defect  free"  and  used  in  the 
inspection  program  to  lower  the  expectation  value  of  the  inspectors 
to  at  least  50%. 

Specimen  Identification 

The  markings  on  each  specimen  should  be  firmly  attached 
or  imprinted  on  the  specimen  in  a manner  so  as  to  orient  the 
location  of  the  flaw  in  the  flawed  specimens,  as  well  as  identify 
the  unflawed  specimens.  The  markings  on  each  flawed  and/or 
control  specimen  should  be  changed  for  each  inspection  batch  dis- 
tribution so  that  inspectors  will  not  be  biased  by  the  recognition 
of  an  individual  specimen  identification. 


7.3  MANAGEMENT  AND  DOCUMENTATION 


Since  the  results  of  the  demonstration  program  are 
to  be  used  in  management  decisions  regarding  the  acceptance 
or  rejection  of  the  NDI  processes,  procedures  must  be  carefully 
monitored  and  documented.  The  management  determines  which  type 
of  inspection  is  to  be  evaluated,  i.e.,  production  line  (in- 
plant  inspection),  production  line  (subcontractor  inspection), 
field  service  inspection,  etc.,  and  also  what  constitutes  a 
failure  of  the  inspection  process. 

It  is  important  that  the  person  or  persons  who  conduct 
the  NDE  demonstration  be  identifiable  in  some  manner.  This  is 
to  ensure  that  subsequent  components  will  be  inspected  by  those 
persons  who  had  contributed  to  the  passing  of  the  demonstration 
program,  and  that  unqualified,  or  marginally  qualified  personnel 
will  not  be  lumped  into  the  inspection  group.  The  purpose  of 
the  demonstration  program  is  not  to  gtade  or  establish  pass-fail 
capabilities,  but  to  ensure  that  the  required  flaw  size  detection 
capabilities  as  established  be  directly  related  to  actual  compo- 
nent inspections.  The  inspectors  should  be  thoroughly  briefed 
on  the  inspection  procedures,  and  periodically  reexamined  in 
normal  operation. 

Surveillance  of  the  NDT  procedures,  and  compliance  by 
both  the  Air  Force  SPO  and  company  management  is  necessary. 

A complete,  extremely  detailed  and  unambiguous  inspection 
procedure  document  should  be  prepared  and  be  available  for  the 
inspector  to  follow.  Clearly  written  instructions  are  necessary 
to  the  success  and  repeatability  of  the  inspection  program.  In 
some  cases  it  may  be  important  to  separate  out  individual  inspectors 
to  determine  the  capabilities  of  different  levels  of  training  and 
experience,  in  other  cases,  it  may  be  necessary  to  evaluate  the 
capability  of  the  company,  grouping  all  inspectors  together. 

Process  control  personnel  should  prepare  the  inspection 
procedure  documents  for  use  in  the  production  inspection.  These 
procedures  should  be  developed  using  identical  specimens  to  those 
used  in  the  actual  demonstration  program,  and  should  not  be  used 
for  any  other  purpose  than  to  prepare  the  required  document. 

The  evaluation  of  these  specimens  for  use  in  the  production  environ- 
ment must  include  cleaning  procedures,  as  well  as  determination 
if  the  specimens  can  be  reused. 
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Specimens  containing  flaws  such  as  fatigue  cracks 
can  be  reused  for  eddy  current,  x-ray  and  magnetic  particle 
and  ultrasonic  inspection  after  proper  cleaning.  For  evaluation 
of  reuseability  of  penetrant  specimens,  a controlled  experimental 
investigation  is  necessary  to  determine  the  number  of  times  that 
the  specimen  may  be  reused. 

A list  of  the  equipment  used  in  the  demonstration 
program  will  be  prepared,  as  well  as  sufficient  information 
as  to  important  dial  settings  and  interpretive  accept-reject 
instructions.  The  inspection  procedure  should  be  prepared 
using  specified  equipment,  and  this  equipment  or  a demonstrated 
equivalent  component,  should  be  explicitly  spelled  out  in  the 
inspection  instructions.  Calibration  procedures  for  verification 
of  dial  settings,  etc.,  should  te  included  in  the  documentation. 

The  format  for  the  reporting  and  identification  of  all 
flawed  specimens  shall  be  prepared  by  the  process  control  per- 
sonnel. These  reports  should  be  in  the  routine  format  for  the 
inspection  group  so  that  undue  attention  is  not  drawn  to  the 
nature  of  the  demonstration  specimens. 

If  at  the  completion  of  the  inspection  program,  destructive 
testing  is  necessary  to  verify  tie  form,  size  and  location  of 
defects  in  the  sample  group  used  in  the  production  inspection, 
these  specimens  should  first  be  re-examined  nondestructively  by 
the  process  control  personnel  in  the  laboratory.  All  specimens 
should  be  examined  at  the  completion  of  the  program  to  verify 
the  actual  size,  shape  and  location  of  defects  for  analysis  in 
the  demonstration  program.  Selected  specimens,  agreed  upon  by 
all  management  and  technical  persons  involved  should  be  destructive 
tested  to  assure  the  control  and  flaw  production  procedures  are 
consistent  with  the  intended  flaw  size  and  shape. 

7.4  INSPECTION  AND  ANALYSIS 

To  evaluate  the  POD  of  90%  at  a 95%  confidence  factor, 
a minimum  of  29  successes  in  29  trials  is  necessary.  This  means 
that  29  specimens  containing  defects  must  be  identified  as  having 
defects  by  the  inspector  to  guarantee  a minimum  of  90/95. 

Random  mixtures  of  control  and  flawed  specimens  should 
be  presented  to  the  inspection  group  for  evaluation.  Approxi- 


mately  25  specimens,  per  inspector,  per  day  for  3 days  would 
be  appropriate.  This  would  result  in  a total  of  75  inspections 
of  both  flawed  and  unflawed  specimens,  in  any  combination 
necessary  to  evaluate  at  least  29  flawed  specimens. 

The  specimens  should  be  cleaned  and  reinspected  the 
following  day  if  necessary  until  sufficient  data  on  flawed 
specimens  is  obtained.  Although  a ratio  of  flawed  to  control 
of  1:1  is  recommended,  this  should  be  considered  as  an  absolute 
minimum,  since  normal  inspection  would  not  reveal  the  presence 
of  so  many  defective  components  in  each  batch.  It  will  become 
obvious  to  the  inspector  that  these  demonstration  specimens 
are  not  of  the  usual  sort  when  he  finds  a large  number  of 
defects,  but  at  no  time  should  the  emphasis  be  made  that  "he" 
is  being  tested,  instead,  the  NDT  evaluation  should  be  emphasized. 
Although  few  human  factor  data  sets  exist  in  completed  form, 
examination  of  individual  data  sets  indicated  that  human  factor 
is  an  important  component  influencing  the  POD.  Although  a 
minimum  of  29  observations  of  flawed  specimens  is  necessary  for 
the  90/95  accept  criterion,  it  is  strongly  suggested  that  addi- 
tional specimens  be  fabricated  and  inspected  in  the  event  that 
some  of  the  specimens  later  verified  as  to  crack  length  have 
actual  lengths  that  fall  outside  of  the  2c  value  selected  for 
verification. 

If  the  initial  inspection  program  does  not  result  in 
verification  of  the  inspection  capability  due  to  a miss  of  a 
flawed  specimen,  decisions  must  be  made  to  continue  the  test 
program  or  to  consider  the  NDT  process  as  less  sensitive  than 
anticipated(H) . As  outlined  in  the  ASM  article  in  Reference  11, 
the  following  combinations  of  successes  and  trials  can  be  used 
to  verify  the  90/95  POD: 


29  successes 

out  of  29  trials 

45 

46 

59 

61 

72 

75 

85 

89  . 

An  inspection  sequence  in  which  no  misses  are  found  is  the 
simplest  and  the  least  costly.  However,  other  factors  may 
influence  the  decision  to  terminate  or  continue  the  inspection 
program  beyond  the  first  29  observations. 
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When  the  total  number  of  observation  of  flawed  specimens 
is  small,  i.e.,  30  or  less,  the  exact  binominal  distribution 
should  be  used  to  determine  the  lower  bound  probability  of 
detection.  For  large  numbers  of  observations,  the  binominal 
distribution  may  be  approximated  by  a Poisson  distribution. 
Procedures  for  calculating  the  exact  value  of  the  lower  bound 
POD  are  given  in  reference  11.  If  an  assessment  of  the 
detection  threshold  to  a finer  degree  is  needed,  the  optimized 
probability  method  should  be  used. 

The  evaluation  of  false  or  error  calls  should  be  noted 
but  not  necessarily  calculated  or  included  in  the  analysis. 

At  the  present  time,  the  objective  of  the  inspection  demon- 
stration is  to  determine  the  ability  to  detect  a defect  given 
the  fact  that  the  defect  is  present,  and  the  hypothesis  does 
not  include  at  present  judgments  about  the  error  call  values. 
One  suggested  procedure  for  evaluation  of  false  calls  is  the 
conditional  probability  method  (12),  but  more  attention  must 
be  given  to  the  magnitude  of  the  weighting  factors  before  this 
can  be  incorporated  into  the  demonstration  program.  Since 
the  false  call  data  is  recorded,  at  subsequent  times  this 
evaluation  can  be  made. 


7.5  VALIDATION  OF  TRANSLATION  MODEL 
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The  design  of  an  optimum  demonstration  program  can  be  used 
to  conduct  an  experimental  program  to  validate  the  translation 
model  developed  for  translating  NDE  results  obtained  on  flat 
plate  specimens  to  specimens  with  complex  geometries.  The  general 
philosophy  and  guidelines  of  such  an  experimental  validation 
program  are  defined  in  this  section.  As  discussed  in  the  previous 
section  on  specimens  to  be  used  in  an  optimum  demonstration  program, 
flat  4x8  in.  specimens  containing  fatigue  cracks  with  controlled 
a/2c  ratio  and  an  appropriate  distribution  of  flaw  sizes  will  be 
used  as  representative  of  specimens  with  simple  geometries. 

Realistic  aircraft  components  will  be  used  to  represent  specimens 
with  complex  geometries.  Fatigue  cracks  with  flaw  sizes  and 
distributions  equivalent  to  those  on  the  flat  plate  specimens  will 
be  introduced  in  the  fillet  areas  of  the  components.  Inspections 
on  both  types  of  specimens  with  simple  and  complex  geometries  will 
be  conducted  using  the  same  NDE  parameters.  The  inspection  results 
will  be  analyzed  by  using  the  adaptive  learning,  linear  regression 
or  point  estimate  techniques.  The  comparison  of  results  for  the 
two  types  of  specimens  will  provide  a basis  for  the  validation  of 
the  translation  model. 

The  results  obtained  from  the  on-going  program  "Quantitative 
Evaluation  of  Penetrant  Inspection  Materials  and  Procedures"  could 
also  be  used  as  baseline  data.  The  purpose  of  using  these  data  is 
two- fold:  (1)  they  can  be  used  to  check  the  validity  of  the  para- 
metric relationship  pertained  to  penetrant  inspection,  and  (2)  they 
can  be  used  as  flat  plate  data  to  develop  a translation  model 
under  the  prescribed  conditions.  The  translation  model 
generated  by  these  additional  data  can  be  compared  with  the 
corresponding  one  developed  under  this  program. 

The  validity  of  the  translation  model  can  be  further  tested 
by  using  existing  aircraft  components  under  damage  tolerance  tests 
and  full  scale  fatigue  tests.  Several  advanced  aircrafts  have 
components  under  these  types  of  tests.  Realistic  fatigue  cracks  are 
expected  to  develop  in  the  critical  areas  in  these  components  after 
several  lives  of  the  fatigue  tests.  Using  these  specimens  and  the 
optimum  demonstration  program  procedures,  production  inspection  can 
be  conducted  with  the  five  NDE  techniques.  The  NDE  capability  will 
be  evaluated  on  these  specimens  with  complex  geometries  and  compared 
with  that  predicted  by  the  translation  model.  The  outcome  of  the 
validation  program  will  be  a set  of  military  standards  such  as 
MIL-I-83444  which  outlines  the  details  to  qualify  NDE  facilities 
to  meet  the  structural  integrity  requirements  in  advanced  aircraft. 
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SECTION  VIII 


CONCLUSIONS 


The  objectives  of  this  program  were  to  establish  NDE 
parametric  relationships , to  develop  a translation  model,  and 
to  design  an  optimum  demonstration  program  eased  on  the  para- 
metric relationship  and  translation  model. 

The  existing  NDE  reliability  data  base  was  enlarged, 
updated,  and  modified  to  serve  as  a basis  for  the  parametric 
study  and  translation  model  development.  Additional  data 
were  entered  into  the  data  bank  for  specimens  with  com- 
plex geometries. 

Analytical  methodologies  in  point  estimate,  linear 
regression  techniques,  and  adaptive  learning  techniques  were 
developed  to  analyze  the  reliability  data.  Computer  programs 
were  coded  to  establish  probability  of  detection  curves  with 
the  best  statistical  fit  according  to  the  three  schemes. 

Qualitative  parametric  relationships  between  the 
probability  of  detection  and  the  NDE  parameters  was  established 
for  aluminum  specimens  and  the  NDE  techniques  of  ultrasonic, 
penetrant  and  eddy  current.  The  NDE  parameters  studied  were 
limited  to  those  included  in  the  existing  data. 

Translation  models  were  developed  to  translate  the 
probability  of  detection  on  fatigue  cracks  obtained  on  flat 
plate  specimens  to  corresponding  values  for  specimens  with 
more  complex  geometries.  The  scope  and  accuracy  of  the  models 
were  somewhat  limited  by  the  data  base.  Overall,  the  trans- 
lation model  from  aluminum  flat  plate  to  fillet  specimens 
was  most  successful. 

Based  on  the  parametric  relationships  and  the  trans- 
lation model  developed  in  the  program,  an  optimum  demonstration 
program  was  designed  to  be  used  as  a guideline  for  future 
validation  and  NDE  facility  qualification  programs. 
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APPENDIX  A 


SAMPLES  OF  COVER  SHEETS  FOR 
EDDY  CURRENT,  RADIOGRAPHIC, 
MAGNETIC  PARTICLE,  AND 
PENETRANT  TECHNIQUES 
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TABLE  A- 1 RELEVANT  NDE  PARAMETERS  FOR  EDDY  CURRENT  METHOD 
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TABLE  A-2  RELEVANT  NDE  PARAMETERS  FOR  RADIOGRAPHIC  METHOD 
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BHf  AVAILABLE  COPY 


TABLE  A-3  RELEVANT  NDE  PARAMETERS  FOR  MAGNETIC  PARTICLE  METHOD 
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TABLE  A-4  RELEVANT  NDE  PARAMETERS  FOR  PENETRANT  METHOD 
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APPENDIX  B 

OVERALL  LOGIC  AND  CODES  OF  COMPUTER  PROGRAMS 
FOR  THE  ADAPTIVE  LEARNING  ANALYSIS 


T 


OVERALL  LOGIC  OF  COMPUTER 
PROGRAMS  FOR  THE  ADAPTIVE  LEARNING  ANALYSIS 

The  adaptive  analysis  of  the  General  Dynamics  data  base 
requires  three  programs: 

1.  Training  and  Testing  Set  Formation  Program.  This 
program  performs  three  functions: 

a)  It  selects  out  of  the  total  data  base  those  data 
sets  that  are  to  be  analyzed. 

b)  It  splits  the  selected  data  base  items  into  two 
sets,  a training  set  and  a test  set. 

c)  It  normalizes  all  variables  in  the  training  and 
testing  sets  to  the  range  -3  to  +3  (approximately). 

2.  Hypersurface  Fitting  Program.  The  function  of  this 
program  is  to  fit  the  training  set  with  a (nonlinear) 
multinomial.  The  fitting  ’procedure  uses  a guided, 
accelerated,  random  seai-ch  with  reversal  and  avoids 
overfitting  through  the  use  of  the  training-testing 
paradigm. 

3*  Analysis  Program.  This  program  is  used  witn  the 
fitted  multmcmials  to  determine  the  relative  con- 
tribution of  each  independent  variable  to  the  output 
variable  (e.g.,  the  probability  of  detection). 

Flowcharts,  mathematical  equations,  and  other  data  for 
these  programs  are  presented  in  this  appendix. 


1.  Definitions 


Variables  and  Arrays 


A DFLT  ( m , n , 6 ) : 


ACUTU?  (m  ,n  ,6  ) : 

BTDTR  (m ,n ,6 ) : 

BTDTE  (m,n ,6 ) : 

XTR  (m,2 ,n.  ) : 
XTE  (m,2,ni): 
XIN  (m ,2 ) : 


the  difference,  ta. , between  the  current 
value  of  a given  network  coefficient,  a^, 
and  a,  .o,  the  best-to-date  value  of  a^. 

The  array  size  variables  refer  to  the1 
number  of  rows  in  the  network,  m,  the  number 
of  columns  in  the  network,  n,  and  tne  number 
of  constants  used  in  each  element,  6. 
the  current  trial  values  of  the  net  coef- 
ficients. 

the  best-to-date  coefficients  found  for 
the  net  using  the  training  set. 
the  best-to-date  coefficients  found  for 
the  net  using  the  test  set. 
the  training  set.  m=NROWS  and  n,=NTR. 
the  test  set.  m=NROVVS  and  n =NTE. 
a temporary  storage  that  holds  the  net 
input  and  subsequent  column  outputs;  used 
in  calculating  net  output;  in  general,  the 
net  input  will  be  the  k-th  subarray  of  XTR  or 
ytp  . 
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YNETR(n.) :the  net  outputs  computed  using  each  of  the 
n,  subarrays  of  XTR. 

YNETECn^ ) :tne  net  outputs  computed  using  each  of  the 
n , sub8rrays  of  XTE. 

YTR(n, ):  the  dependent  variable  subarray  of  the  train- 

ing set . 

YTF(n2):  the  dependent  variable  subarray  of  the  test 

set . 


Array  Size  Parameters 


NTR : 

the 

number 

of 

NTE: 

the 

number 

of 

NVAR: 

the 

number 

of 

XTR 

or  XTE 

NNFT: 

the 

number 

of 

NROWS: 

the 

number 

of 

NCOLS : 

the 

number 

of 

NCOEF: 

the 

number 

of 

(mx2)  subarrays  in  XTR 
(mx2)  subarrays  in  XTE 
independent  variables  in  either 

elements  (modules)  in  the  network 
rows  of  elements  in  the  network 
columns  of  elements  in  the  network 
coefficients  in  the  network 


Options  and  Input  Constants 


PRINT 

IRANN 

NVAR 

NTE 

NTP 

NNFT 

NROWS 

NCOLS 

NCOEF 

Cl 

C2 

C5 

C4 

NSTOP 

AMAX 


0 if  intermediate  printout  not  desired,  1 if  de- 
sired 

number  used  to  initialize  the  random  number  rou- 
tine (allows  exact  duplication  of  a run  if  desired) 
input  from  training/testing  set  formation  program 
input  from  training/testing  set  formation  program 
input  from  training/testing  set  formation  program 
NROWS *NCOLS 

NVAR/2  for  a rectangular  or  square  net 
NROWS  in  a square  net 

6*NNET  if  a six-term  net  element  is  used 
smoothing  constant;  set  to  1.0 
smoothing  constant;  set  to  10.0 
smoothing  constant;  set  to  n 
smoothing  constant;  set  to  20 

trial  number  for  unconditional  search  halt;  set  to 
1000 

parameter  size  constant;  set  to  10.0 


Miscellaneous  Constants 

SZFRO  = smoothing  constant  computed  during  initialization 


Indexes  and  Pointers 

I = trial  number 

J = number  of  times  step  size  doubles  during  hill-climbing 


r 


KTR  = k-th 
KTF  = k-th 
KVAR  = k-th 
KNET  = k-th 
KROW  = k-th 
KCOL  = k-th 
KCOFF  = k-th 


training  set  member 
test  set  member 

variable  in  a training  or  testing  set 
module  (element)  in  the  network 
rov;  of  the  network 
column  of  the  network 
coefficient  out  of  the  6*NNFT  total 


vector 


Subroutines 

RANI:  generates  a random  number,  a,  from  a flat  dis- 

tribution; 0 < a < 1 

RAN2:  generates  a number  from  a zero-mean,  unit- variance 

distribution;  fi 

AMFND(m,n ,6 ) : checks  each  random  number  produced  by  RAN2 
to  see  if  any  have  gone  outside  the  preset  boun- 
dary, +a  . If  any  have  gone  outside,  AMEND 
computesmlxnew  value  of  ACURR  which  is  within 
limits  and  also  corrects  ADELT 

YELT:  computes  the  output  of  a network  element,  given 

the  input  and  coefficient  vectors 

YNETR(NTR) : computes  the  net  output  for  each  subarray  of 
the  training  set,  given  a coefficient  array 

YNETF(NTF):  computes  the  net  output  for  each  subarray  of 
the  test  set,  given  a coefficient  array 

SCORR : computes  the  sum  of  the  squares  of  the  two  ( lxNTR ) 
arrays  YTR  and  the  current  YNFTR  and  divides  by 
NTR 

SCORE:  computes  the  sum  of  the  squares  of  the  two  (lxNTE) 
arrays  YTF  and  the  current  INETE  and  divides  by  NTE 


Print  Messages  Used  in  Fitting  rrogram 
Header  Block: 

Print  program  name,  run  number,  and  options  and  data  by 
variable  name  and  value.  If  PRINT  = 1,  set  up  column 
headings  for  intermediate  printout: 


Trial  No. 

Search  Mode 

Score , Tr 

Delta  S 

Score,  Te 

NBTD 

(Message  #1): 

I 

"un guided" 

SBTDR 

SDELT 

SBTDE 

NBTDE 

(Message  #2): 

I 

"guided" 

SBTDR 

SDELT 

SBTDE 

NBTDF 

(Message  #5): 

I 

"reversal" 

SBTDR 

SDFLT 

SBTDE 

NBTDE 

(Message  #4): 

I 

"8ccel" 

SBTDR 

SDELT 

SBTDE 

NBTDE 

Final  Message: 

"Score,  Tr"  "Score,  Te" 


JV/Ui  c • 

SHTDR 


SBTDE 


"Best  Test  Trial" 
NBTDE 
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A 


Coefficient  Set 


ABTDF(1,1 ,1)  ABTDE( 1 ,1,2) 
ABTDE(2 ,1,1)  ABTDF(2 ,1,2) 

• • • 

ABTDF(n,n,l)  . . . 


ABTDE( 1 ,1,6) 
ABTDE(2 ,1,6) 

ABTDE(n ,n  ,6  ) 


2 . Training  and  Testing  Set  Formation  Program 

Data  must  be  selected  from  the  General  Dynamics  data  base 
and  transferred  to  an  intermediate  stroage  which  is  accessible 
by  the  fitting  and  analysis  programs.  Selection  is  required 
since  only  a portion  of  the  total  data  base  is  to  be  fitted  at 
any  one  time.  It  appears  likely  that  the  existing  selection 
program  used  with  the  data  base  could  be  compatible  «.ith  the 
requirements  of  the  adaptive  analysis,  and  the  decision  as  to 
whether  to  use  the  existing  program  or  prepare  a new  one  is 
left  to  the  project  programmer. 

The  division  of  data  points  to  either  the  testing  or 
training  set  should  be  done  on  a random  basis,  although  a syste- 
matic basis  such  as  odd-numbered  points  to  be  assigned  to  the 
testing  set  and  even-numbered  points  to  the  training  set  would 
probably  not  introduce  systematic  biases  to  the  sets  in  view  of 
the  independence  of  the  measurements  in  the  data  base.  It  is 
recommended  that  the  odd-even  assignment  rule  be  used  to  divide 
the  data  points  into  training  and  testing  sets. 

Each  training  and  testing  set  is  comprised  oi  a set  of 
measurement/observation  vectors  on  specimens,  together  with  the 
dependent  variable  that  corresponds  to  the  particular  vector. 
E.g.,  let  X ^ be  the  i-th  vector;  then 

Xi  = ^X1 ,x2  * ‘ ' ,xn^i 

where 

x^  = crack  identification 
x^  = crack  length 
x^  = crack  depth 
x^  = surface  finish 


The  t-t  sets  will  consist  of  some  number,  m,  of  such 
vectors  and  corresponding  dependent  variables,  : 

T = »yi  ) * ’ •••  ’ (Xm’^m^ 

The  number  of  elements  in  the  training  and  testing  sets  need  not 
be  equal,  but  they  should  be  approximately  the  same.  The  minimum 
value  for  m probably  should  be  about  20,  but  an  exact  value  can 
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oply  be  determined  by  trial  and  error.  It  is  important  to  note 
that  if  any  of  the  independent  variables,  x.  , is  constant  for 
all  m (in  both  the  training  and  testing  sets),  it  should  be 
deleted  from  these  sets  since  it  cannot  affect  the  fit. 

For  technical  reasons,  it  is  desirable  to  rescale  all 
members  of  the  training  and  testing  sets  and  to  organize  them 
into  three-dimensional  arrays  instead  of  two-dimensional  arrays. 
The  independent  variables  in  the  original  arrays  should  be  di- 
vided into  two  groups,  e.g.,  the  even  and  odd  variables.  If 
the  number  ol  variables  is  odd  to  begin  with,  it  can  be  made 
even  by  defining  a new  variable  whose  value  is  zero.  As  dis- 
cussed above,  adding  an  unchanging  variable  does  not  afiect  the 
fit  of  the  hypersurface.  After  reorganization,  the  sets  should 
be  an  (m,2,n, ) and  an  (m,2,n,)  array,  where  m is  now  the  number 
of  rows  in  tne  net  instead  or  the  number  of  variables.  Also,  the 
dependent  variable,  y,  , should  be  split  off  to  form  a YTR(n,)  and 
a YTE(np)  array.  This  three-dimensional  array  simplifies  the 
later  network  calculations. 

The  rescaling  is  done  for  all  samples  of  the  dependent 
and  independent  variables.  The  procedure  is  to  calculate  the 
mean  and  standard  deviation  of  all  variables  in  the  testing  and 
training  sets  combined,  then  subtract  off  the  mean  of  each  varia- 
ble and  divide  by  the  standard  deviation.  This  scaling  brings 
all  variables  into  approximately  the  -3  to  +t>  range,  which  im- 
proves the  probability  that  acceptable  solutions  for  the  coef- 
ficients will  lie  in  the  range  -amgx  to  +amgx,  where  amax  is 
about  10. 


3 . Fitting  Program 

This  program  accomplishes  the  task  of  finding  the  best 
set  of  coefficients  for  the  multinomial.  The  fitting  program 
uses  a guided,  accelerated,  random  search  with  reversal,  ter- 
minal search  smoothing  and  overfitting  control.  The  basic  sub- 
routines in  this  program  are  used  over, and  over  as  can  be  seen 
in  tne  flowchart  for  the  program  (Figure  B-l).  Each  subroutine 
will  be  described  in  some  detail. 


3-1  Random  Number  Generators 

The  two  random  number  generators  are  RANI  and  RAN2. 

RANl(n):  Generates  n random  numbers  drawn  from  a rectan- 

gular distribution,  0<a<l.  These  numbers 

are  then  scaled  to  the  -a__„  to  + a „ range 

max  max 

using  a.nax(2a.  - 1).  When  X.vXX(n)  = RANl(n) 
is  written,  it  is  to  be  interpreted  as  calling  for 
the  generation  of  n scaled  random  numbers  and 
replacing  the  array  XXXX  by  this  random  array. 
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START 


SET 

BTDTR  (NCOEF) 

= RAN  1 (NCOEF) 


DO  YNETR; 
GIVEN  XTR, 
BTDTR 


DOSCORR; 
GIVEN  YTR, 
YNETR  SET 
SZERO  = SCORR 


Figure  B-1  Hypersurface  Fitting  Program  Initialize  and  Load  BTDTE  and 
SBTDE 


i 

H 

A 


132 


133 


2 


J = 0 


00 

AOELT  (NCOEF) 
= RAN  2 (NCOEF) 
DO 

AMEND 


BTDTR  = ACURR 
SBTDR=SCURR 


DO 

ACURR  (NCOEF) 

= BTDTR  (NCOEF) 
+ ADELT (NCOEF) 


DO  YNETR; 
ACURR, 


DO  SCURR  = 
SCORE; 
YNETR,  YTR 


00 

SDELT  = 
SCURR-SBTDR 


PRINT 
MESSAGE 
NO.  2 


YNETE, 

USE 

BTDTR, 

YTE 


SCURE  = SCORE; 
USE 

YNETE,  YTE 


BTDTE  = BTDTR 
NBTDE  = I 
SBTDE  = SCURR 


<5 


Figure  B-1  Hypersurface  Fitting  Program  (Continued)  Guided  Search  and  t-t  Paradigm 
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Figure  B-1  Hypersurface  Fitting  Program  (Concluded)  Reversal 
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RAN2(n):  Generates  n pseudorandom  numbers,  p,  from  a 

zero-mean,  unit-variance  distribution  and 
scales  them  to  the  proper  range.  If  written 
as  XXXX(n)  = RAN2(n),  it  is  interpreted  to  mean 
compute  n random  numbers  and  use  them  to  replace 
XXXX.  The  scaling  on  RAN2  is: 

output  = (1/3)(C1)(^)(SBTDR/SZER0)C5/(1  + I/C2) 

Good  starting  values  for  Cl,  C2,  C3  and  C4 , 
respectively,  are  1,  10,0,  k>. 

The  RAN2  routine  must  always  be  followed  by 
the  AMEND  routine 

Generally,  both  RANI  and  RAN2  are  available  in  a single 
standard  software  package  with  most  computers.  If  it  is  desired 
to  start  a run  with  a particular  random  number  (which  then  uniquely 
determines  all  successive  numbers  generated)  it  is  possible  to 
initialize  these  special  software  packages  with  a fixed  random 
number,  and  a test  must  be  inserted  in  the  program  immediately 
after  initialization  of  the  arrays.  The  test  would  be  applied 
to  the  constant  IRANN : if  IRANN  = 0,  do  not  initialize  the 
random  number  generator;  if  IRANN  *0,  initialize  the  random 
number  generator  with  IRANN. 


3.2  AMFND 

This  subroutine  checks  the  coefficient  vector  generated 
by  RAN2  or  by  reversal  to  see  if  any  of  the  coefficients  are 
outside  the  preset  boundary.  The  basic  algorithm  employed  can 
be  thought  of  as  a reflecting  barrier  at  the  boundary,  and  if  a 
given  carries  the  i-th  variable  to  a value  a + , the 

position  of  a.  is  reset  to  a.  - . Since  it  is  theoretically 
possible  for  to  be  larger  than  2a  , it  is  necessary  to  repeat 
the  test  after  adjustment  to  be  sur§  that  the  final  a.  is  in 
limits.  This  subroutine  must  always  follow  ADELT  computed  from 
PAN2,  from  ADELT  = 2*ADELT,  and  fromADFLT  = -ADELT.  It  is  not 
needed  for  RANI  programs,  since  the  variables  are  always  in 
limits  by  construction,  nor  is  it  ever  used  with  coefficients 
used  in  computing  test  set  values  since  no  aa's  are  ever  com-, 
puted  for  the  test  set  coefficients. 


3.3  Scoring:  SCORE  and  SCORR 

The  SCOR  subroutine  computes  the  average  sum  of  squares 
of  the  errors  in  hypersurface  fitting.  Since  scores  are  needed 
for  both  training  sets,  SCOR  is  always  postscripted  with  an  R 
or  an  E to  distinguish  between  the  two  scores.  The  formula  for 
SCORR  is 

scom  '("Ki 
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where  a and  yS.  are  dummy  variables  indicating  the  i-th  members 
of  YTR  ind  YNE^R,  respectively. 

Similarly,  SCgRE  is  given  by 

SCORE  = - 2 (a -fi.)2 

2 i=l 

and  the  dummy  variables  are  members  of  YTE  and  YNETE. 


3.4  Network  Output  Computation:  YNETR  and  YNFTE 

Network  connectivity  is  a pictorial  representation  of  the 
order  in  which  net  inputs  or  intermediate  column  outputs  must  be 
combined  using  the  net  element  algorithm  to  achieve  computation 
of  a multinomial.  There  are  two  connectivity  patterns  that  have 
been  found  to  have  greater  utility:  rectangular  and  exponential. 

The  algorithm  for  the  square  network  is  the  simplest  to  program 
and  has  been  selected  for  the  first  analyses  for  this  program. 

The  elements  in  a net  should  be  organized  into  columns  and  the 
columns  numbered  from  1 on  the  input  side  to  n at  the  output 
side.  Within  columns,  the  elements  should  be  numbered  from  top  to 
bottom.  The  network  input  vector  should  always  be  presented  to 
the  net  with  the  variable  number  in  ascending  order  from  top  to 
bottom  of  the  net.  The  calculation  procedure  to  be  followed  is  to 
compute  the  outputs  of  columns  from  top  to  bottom  (i.e.,  in  order 
of  increasing  row  number).  This  procedure  guarantees  that  the 
intermediate  outputs  will  be  available  when  needed  for  any 
succeeding  computation. 

It  is  clear  that  "network  connectivity"  is  simply  an 
addressing  scheme  as  far  as  the  computer  program  is  concerned. 

The  reorganization  of  the  training  and  testing  sets  into  3-dimen- 
sional arrays,  discussed  earlier,  was  designed  to  simplify  the 
net  addressing  algorithm. 

The  net  element  computes  the  function 

11  = a!  +a2  $ l+a3  $ 2+a4  * l+a5  $ 1 * 2+a6  * 2 

To  expand  an  element  computation  to  a full  net,  the  connectivity 
algorithms  must  specify  the  addresses  of  the  coefficient  vector 
and  the  input  vector.  The  training  end  testing  sets,  which 
provide  the  inputs,  were  organized  as  (m/2,  2,  n^)  and 
(m/2,  2,  n?)  arrays.  Let  (i,j,k)  be  a pointer  to  a specific 
element  in  the  array.  If  k is  held  fixed,  then  x(i,j)  is  an 
(m/2,2)  array,  and  this  array  is  exactly  the  net  input  needed 
for  that  value  of  k.  It  can  be  noted  that  the  index,  i,  in 
x(i,j)  is  always  the  same  numerical  value  as  k, , the  row  index 
for  the  net.  Recall  that  the  number  of  input  variables  was  forced 
to  be  even  to  allow  equating  i and  k-^. 
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The  general  expression  for  net  addressing  requires  four 
indexes:  (i  ,k  ,k  _,)  . The  (i,k^)  pair  give  the  row  and  column 
of  the  net  element;  the  index,  j,  distinguishes  between  the 
l and  2 inputs,  and  k refers  to  the  k-th  member  of  the  train- 
ing set.  Since  the  XIN  array  is  used  only  for  the  first  column 
calculation,  it  can  serve  as  temporary  storage  for  succeeding 
column  outputs.  To  show  this,  let  y(i,j,k,k2)  be  an  element 
output.  Then  y can  be  written  into  x(i,j,k)  since  there  will 
always  be  twice  as  many  x's  as  y's.  Once  all  m/2  outputs  for 
a column  have  been  computed,  then  set 

x(l,2,k)  = x(f,l,k),  i=l 

x (i ,2 ,k ) = x(i-l ,1 ,k) , i=2,...,m/2 

The  input  array  to  the  next  column  is  then  set  up,  exactly  as 
the  net  inputs  set  up  the  first  column  inputs.  It  should  be 
noticed,  however,  that  this  algorithm  is  equivalent  to  a net 
with  a uniform  pitch  of  1.  If  another  pitch  pattern  is  desired, 
the  above  two  formulas  would  have  to  be  modified. 

The  net  output  can  be  computed  once  the  final  column 
calculation  has  been  made: 

m/2 

y(k)  = (2/m)  >:  x(i  ,1  ,k) 

i = l 

To  distinguish  between  outputs  computed  for  the  training 
and  testing  sets,  the  notation  YEETR  means  that  the  net  output 
is  computed  for  the  training  set  and  YE’ETE  means  that  the  net 
output  has  been  computed  for  the  test  set.  In  either  case, 
however,  it  is  still  necessary  to  indicate  the  origin  of  the 
coefficients  used  in  the  calculation. 

The  network  element  coefficients  used  in  the  calculations 
are  the  6(m/2)^  elements  in  one  of  the  arrays  ACUPE,  BTDTF  or 
B7DTR.  The  indexes  for  the  a-arrays  are  (i,kj,k-„. ),  k = l,...,6, 
and  the  successive  1x6  arrays  of  coefficients  are  addressed 
directly  to  the  proper  elements. 


4 . Analysis  Program 


The  analysis  program  approximates  the  first  derivative  of 
y,  the  net  output,  with  respect  to  each  variable,  x. , in  the 
training  cr  testing  set  using  first  difference  formulas: 


where 


y(X  + mx 

uX 


1 if  j = i 
0 if  3 / i 
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and 


X 


(x1 , . . . ,xi,  . . . 


y(X)  is  computed  using  the  net  subroutine 


This  calculation  should  be  performed  for  each  variable 
at  each  point  in  the  training  set.  This  procedure  will  produce 
NTP  values  for  each  partial  derivative.  A useful  way  to  plot 
the  results  of  this  program  is  as  a scatter  plot  on  a /ry/mx. 
vs.  x.  graph,  and  it  is  very  helpful  to  organize  the  (x,  ,my/mx . ) 
pairs1in  ascending  order  of  the  x-  vuJues.  Since  this  procedure 
is  used  for  each  variable,  there  will  be  a total  of  NV/tPxNTP 
points  to  be  plotted,  NTP  for  each  variable. 

Since  all  x.  values  lie  approximately  in  the  -5  to  +5 
range  after. normalization , a reasonable  value  for  ux  would  be 
about  6xlO-;5,  which  is  1/1000  the  range  of  the  x's.  Since  the 
mean  and  variance  for  each  variable  were  computed  in  the  training 
and  testing  set  preparation  program,  the  partials  and  the  x's 
could  be  scaled  back  to  the  original  values  if  the  means  and 
standard  deviations  were  preserved. 

Using  only  a first  difference  analysis,  it  is  possible 
to  eliminate  those  variables  which  make  no  significant  contri- 
bution to  y (those  for  which  all  the  points  on  the  scatter  plot 
lie  very  close  to  zero).  A more  detailed  analysis  can  be  ob- 
tained by  taking  only  the  significant  or  contributory  variables 
and  computing  various  mixed  partials  or  by  cross-plotting  to 
obtain  a family  of  curves  with  some  of  the  other  variables  as 
parameters  of  the  plots. 

Once  the  significant  variables  have  been  determined,  a 
smaller  net  can  be  fitted  using  only  these  variables.  If 
SBTDF  for  the  refitted,  smaller  net  is  within  about  5%  of  its 
value  in  the  larger  net  in  which  all  variables  were  present, 
then  the  smaller  net  can  be  used  for  the  more  detailed  studies 
and  cross-plotting. 
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COMPUTER  CODES  FOR  THE  ADAPTIVE  LEARNING  ANALYSIS 

The  basic  program  consists  of  a main  resident  program  and 
eight  overlays.  The  main  program  resides  in  core  at  all  times 
and  calls  in  the  overlays  by  the  routine  LINK  as  needed.  Figure 
B-2  presents  a listing  of  the  main  program. 

Overlay  1 

The  function  of  Overlay  1 is  to  select  NDE  method,  enter 
parameters  to  sort  on,  with  upper  and  lower  limits  as  an  option, 
select  data  base  and  select  the  net  input  variables. 

Figures  B-3  through  B-7  show  the  teletype  printout  from 
the  overlays.  The  first  response,  "TITLE  IS:",  is  a request  for 
title.  The  second  line  prints  out  the  title  along  with  the  date 
and  time.  Next  the  NDE  method  is  specified.  The  next  teletype 
response  is  requesting  the  selection  of  NDE  parameters  to  sort 
on.  These  parameters  must  be  indicated  as  numeric  coded  numbers. 
For  operator  convenience  the  available  parameters  for  a given 
NDE  method  along  with  their  numeric  codes  are  displayed  on  the 
Tektroni-X4010  scope.  The  teletype  next  responds  with  "IS  THIS 
OK?",  giving  the  operator  an  opportunity  to  change  the  para- 
meters. The  next  teletype  response  asks  for  limits  on  para- 
meters and  then  responds  with  "IS  THIS  OK?",  giving  the  operator 
a chance  to  change  the  limits.  The  teletype  then  responds  with 
"ENTER  DATA  SET  # FROM  KEY  BOARD?".  This  is  an  option  to  either 
enter  the  daca  sets  from  the  key  board  or  let  the  computer  search 
through  the  disc  and  locate  the  data  sets  which  match  the  given 
control  lines.  After  the  data  base  has  been  specified,  the  tele- 
type responds  with  "MORE  CONTROL  LINES?".  This  is  an  option 
which  permits  the  operator  to  select  a different  set  of  para- 
meters. After  the  operator  responds  to  this  option  the  teletype 
responds  with  "NET  INPUT  VARIABLES".  The  operator  must  respond 
with  the  desired  net  input  variables.  The  order  in  which  the 
variables  are  specified  will  determine  the  order  that  the  variables 
are  fed  into  the  network. 
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C AL.FTN  , GD/FV/MTL  , MAY  2,1976 
C ADAPTIVE  LEARNING  PFOGTAM 
C FESIDENT  OVEPLAY 

COMMON  / APFAYS  / I TL ( 30 ) , K LL C 2, 8 ) , N AM E( 1 0, 1 2 ) , 

1 LY<  1 6), LI  SC  100), AC  17),  SDC  17), BTEC6, 36), 

2 YTPC 100), YTEC 100),IYTFQC100),IYTEQC100> 

COMMON  / CONTFL  / MO  LE,  N LS,  I SC,  I PEC  1 , I PEC  2, 

1 MTHL,NTF,  NTE,  N VAP,  NNET,  NFOW,  NCOL,  NCOEFF  , 

2 API  AS,  Cl,  C2,  C3,  C4,  AMX,  SZEPO,  I DM  ( 10),XDMC  5) 

C DEFINE  DIPECT  ACCESS  FILES 

DEFINE  FILE  1 ( 4500,  256,  U,  I PEC  1 ) 

DEFINE  FILE  2C  2,  1 200,  U,  I FEC2) 

C FOPCE  LOADING  OF  CEPTAIN  FOUTINES  IN  MOS 
X=  2. *X/ ( 1 .+X)-  1 . 

I =X 

C SELECT  DATA  SETS 
1 CALL  LINKC ’Ll’) 

C SELECT  POINTS  F FOM  DATA  SETS 
CALL  L I NK ( ’ L2  ’ ) 

C AFFANGE  DATA  POINTS  AN  L SCAN  THFOUGH  DATA 
CALL  L I NK ( ' L 3 ' ) 

CALL  LINKC 'L4*  ) 

C ENTEP  FITTING  CONSTANTS  , INITIALIZE  COEFF.  AFPAY 
CALL  LINKC ’L5*  ) 

C FIT  THE  DATA 

CALL  LINKC  'L6  ' ) 

C DISPLAY  THE  DERIVATIVES  AND  EVALUATE  THE  FIT 
CALL  LINKC  *L7  ' ) 

C PEPFOPM  A PAPAMETPIC  STUDY  , PLOT  FAMILY  OF  CURVES 
CALL  LINKC 'L8 ’ ) 

GO  TO  1 
END 


Figure  B-2  Adaptive  Learning  Technique  Main  Program  Listing 


BEST  AVAILABLE  COPY 
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TITLE  IS  i PENETRANT  FLAT/ PLATE 

PENETRANT  FLAT/PLATE  19-N0V-76  12:02: SI 

NOE  METHOD  « PENETRANT 

ENTER  PARAMS 

2 I 43 

3 I 45 

4 : 47 

5 « 48 

6 : 0 

IS  THIS  OK  ? YES 
LIMITS  ON  PARAM  s 43 
LOWER.  UPPER  : 1.640 

LIMITS  ON  PARAM  : 47 
LOWER.  UPPER  : 1.10 

LIMITS  ON  PARAM  : 48 
LOWER.  UPPER  s 1.300 

umttc  n m PARAM  ! 0 

IS  THIS  OK  ? YES 

ENTER  DATA  SET  # FROM  KEY  BOARD  ? YES 
1:1 
2:  2 


3:0 

MORE  CONTROL  LINES  7 NO 

1 2 

NET  INPUT  VARIABLES 
1:  43 

:: ::  ©i  availabiecopy 

4:  14 

5:  47 
6:  48 
7:  0 

Figure  B-3  Teletype  Output  from  Overlay  1 
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# VECTORS  : 50 


MIN  POP.  s 2 

MAX  JUMP  SI  ZE=MJ  S , MAX  POINTS  PER  SET=MPPS 
SET  MJS.MPPS  FOR  EACH  DATA  SET  ? NO 

MJ  Sj  MPPSt  2j  60 

IS  THIS  OK  ? YES 

USE  LINEAR  INPUTS  ? NO 

SKIP  CELL-POP.  ? NO 

MIN  POP  TO  INITIATE  CF=  1 ? 9 

INPUT  ICFtl), IPOP  = 1 10 

INPUT  ICF(I),IPOP  = 2 10 

INPUT  I CF( I ) / I POP  = 3 10 

INPUT  I CF< I > .»  I POP  = 4 50 

INPUT  ICF(I)#IPOP  = 5 50 

INPUT  I CF( I ) « 1 POP  = 6 50 

INPUT  I CF( I >.*  I POP  = 7 100 

INPUT  ICF(I),IPOP  = 8 100 

INPUT  I CFC I ) > IPOP  =9  100 

IS  THIS  OK  ? YES 


OVERLAY  2 


OVERLAY  3 


OVERLAY  4 


Figure  B-4  Teletype  Output  from  Overlays  2,  3,  and  A 


NORMALIZE  ? YES 


USE  MEANASTND.  DEV. 

? NO 

CRACK  LENGTH 

Mt 

0.078 

B: 

0.178 

Ss 

SAMPLE  HISTORY 

M: 

0.  197 

Bs 

0.408 

Ss 

CRACK  LENGTH 

Ms 

0.078 

B: 

0.178 

Ss 

PART  GEOMETRY 

M: 

0.000 

Bs 

0 . 000 

Ss 

SURFACE  FINISH 

Ms 

-0.017 

Bs 

0.729 

Ss 

THICKNESS 

M: 

-0.078 

Bs 

0.797 

Ss 

NORMALIZE  Y ? NO 

ZERO  NON-LINEAP  COEFFICIENTS  ? NO 
ZERO  CONSTANT  TERMS  ? YES 
ZERO  BTE  ARRAY  ? YES 
Cl  : . 1 
C2  : 1 0 . 

C3  : . 5 
C4  s 20. 

NCOL  : 3 

SET  CONSTANTS  FOR  <A2,A3>  ? NO 
AMX  : 1 . 

PRINT  INTERVAL  : 150 

USE  RANDOM  NO.  GENERATOR?  NO 

WHICH  NET?  0,0 


Figure  B-5  Teletype  Output  from  Overlay  5 


0.059 

0.109 

0.059 

0.585 

0.117 

0.116 
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BBTAVAIIABLE  COPY 


MODE 

# 

BTD-TRAIN 

BTD- TEST 

NBTD 

UNGU 

l 

0.399E  00 

0.349E 

02 

0.355E  00 

0 

0.0000 

T-T 

4 

0.340E  00 

0.340E 

00 

0.302E  00 

4 

0.0303 

T-T 

5 

0.256E  00 

0.256E 

00 

0.227E  00 

5 

0.0303 

T-T 

6 

0.681E-01 

0.681E- 

01 

0. 515E-01 

6 

0.0303 

T-T 

8 

0.675E-01 

0.675E- 

>01 

0. 506E-01 

8 

0.0087 

T-T 

9 

0.669E-01 

0.669E- 

01 

0.495E-01 

9 

0.0087 

T-T 

I 1 

0 • 650E-0 1 

0.650E- 

'01 

0.459E-01 

1 1 

0.0086 

T-T 

16 

0.635E-01 

0 • 635E-0 1 

0.441E-01 

16 

0.0083 

UNGU 

162 

0.635E-01 

0.799E 

01 

0.441E-01 

16 

0.0082 

UNGU 

302 

0.635E-01 

0.245E 

01 

0. 441 E- 01 

16 

0.0082 

UNGU 

462 

0.635E-01 

0.707E 

00 

0.441E-01 

16 

0.0082 

UNGU 

602 

0.635E-01 

0. 640E 

01 

0.441E-01 

16 

0.0082 

UNGU 

762 

0 • 635E-0 1 

0*  573E 

03 

0.441E-01 

16 

0.0082 

UNGU 

902 

0.635E-01 

0.488E 

00 

0. 441E-01 

16 

0.0082 

UNGU 

1062 

0.635E-01 

0.829E-01 

0.441E-01 

16 

0.0082 

UNGU 

1202 

0.635E-01 

0.277E 

01 

0.441E-01 

16 

0.0082 

UNGU 

1362 

0.635E-01 

0 • 243E 

01 

0 . 44 1 E- 0 1 

16 

0.0082 

UNGU 

1 502 

0.635E-01 

0.274E 

00 

0.441E-01 

16 

0.0082 

UNGU 

1662 

0.635E-01 

0. I82E 

01 

0.441E-01 

16 

0.0082 

UNGU 

1802 

0.635E-01 

0* 1 73E 

04 

0.441E-01 

16 

0.0082 

UNGU 

1962 

0.635 E- 01 

0*  344E 

01 

0.441E-01 

16 

0.0082 

UNGU 

2102 

0. 635E-01 

0. 676E 

00 

0.44  IE- 01 

16 

0.0082 

UNGU 

2262 

0.635E-01 

0.421E 

01 

0.441E-01 

16 

0.0082 

UNGU 

2402 

0.635E-01 

0.382E 

02 

0.441E-01 

16 

0.0082 

UNGU 

2562 

0.635E-01 

0.405E 

01 

0.441E-01 

16 

0.0082 

UNGU 

2702 

0.635E-01 

0*  38  IE 

01 

0.44  IE- 01 

16 

0.0082 

UNGU 

28  62 

0 • 635E-0 1 

0.970E 

00 

0 . 44  IE-01 

16 

0.0062 

UNGU 

3002 

0.635E-01 

0* 1 38 E 

02 

0.44  IE- 01 

16 

0.0082 

UNGU 

3162 

0. 635E-01 

0* 1 24E 

01 

0. 441E-01 

16 

0.0082 

UNGU 

3302 

0.635E-01 

0.61  IE 

01 

0.441E-01 

16 

0.0082 

UNGU 

3462 

0 • 635E-0 1 

0. 103E 

01 

0.44 IE- 01 

16 

0.0082 

UNGU 

3602 

0.635E-01 

0.975E 

00 

0.441E-01 

1 6 

0.0082 

UNGU 

3762 

0.635E-01 

0. 366E 

02 

0.44 IE- 01 

16 

0.0082 

UNGU 

3902 

0*635 E- 01 

0. 365E 

00 

0.441E-01 

1 6 

0.0082 

UNGU 

4062 

0.635E-01 

0. 343E 

01 

0.44 IE- 01 

16 

0.0082 

UNGU 

4202 

0 • 635E-0 1 

0. 1 22E 

02 

0.441E-01 

16 

0.0082 

UNGU 

4362 

0.635E-01 

0. 21 5E 

04 

0.441E-01 

16 

0.0082 

UNGU 

4502 

0 • 635E-0 1 

0. 257E 

02 

0.44  IE- 01 

16 

0.0082 

UNGU 

4662 

0.635E-01 

0.705E 

01 

0. 441E-0 1 

16 

0.0082 

UNGU 

4802 

0.635E-01 

0.339E 

01 

0.441E-01 

16 

0.0082 

UNGU 

4962 

0.635E-01 

0.246E 

0 1 

0.441 E- 0 1 

16 

0.0082 

UNGU 

5102 

0.635E-01 

0.239E 

02 

0.441E-01 

16 

0.0082 

UNGU 

5262 

0.635E-01 

0. 516E 

01 

0.441E-01 

16 

0.0082 

UNGU 

5402 

0.635E-01 

0. 161E 

02 

0.441E-01 

16 

0.0082 

UNGU 

5562 

0.635E-01 

0. 182E 

01 

0.441E-01 

1 6 

0.0082 

UNGU 

5702 

0.635E-01 

0.  1 78E 

02 

0.441 E- 01 

1 6 

0.0082 

UNGU 

58  62 

0.635E-01 

0.  1 70E 

01 

0.441E-01 

16 

0.0082 

UNGU 

6002 

0 * 635E-0 1 

0.31 3E 

02 

0.441E-01 

16 

0.0082 

UNGU 

6162 

0. 635E-01 

0. 1 31E 

02 

0.441E-01 

16 

0.0082 

UNGU 

6302 

0.635E-01 

0.524E 

01 

0.44 IE- 01 

16 

0.0082 

UNGU 

6462 

0.635E-01 

0.418E 

00 

0.441E-01 

16 

0.0082 

UNGU 

6602 

0.635E-01 

0. 1 31E 

00 

0.441E-01 

16 

0.0082 

FIN  I 6705 
FINI  TIME  : 

0* 186E  00 
16:40:25 

0.27  IE 

00 

0. 1 66E  00 

16 

0.0082 

Figure  B-6  Teletype  Output  from  Overlay  6 
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The  subroutines  and  their  functions  for  this  overlay  are 
listed  below. 

OVERLAY  1 SUBROUTINES 

LOOP  - Selects  data  sets  for  data  base 

SELECT  - Selects  parameters  with  upper  or  lower  limits  to 
sort  on 

NDEMTH  - Selects  NDE  method 

PRMTR  - Lists  all  available  parameters  for  a given  NDE 
method 

VRY  - Lists  values  for  each  parameter 

MWWN  - Matches  words  with  numbers 

LGND  - Legend  for  data  file 

FETCH  - Reads  header  page  for  a given  data  set 

MATCH  - Match  given  control  line  with  control  line  from 
disc . 

Overlay  2 

The  primary  function  of  this  overlay  is  to  pick  the  vectors 
from  the  data  base,  each  vector  having  the  same  common  net  input 
variables  as  specified.  The  maximum  number  of  vectors  which  the 
program  can  handle  is  specified  by  two  dimension  blocks  of  600  words 
each.  One  block  is  reserved  for  the  training  set  and  one  for 
the  testing  set. 

The  arrays  for  the  training  (XTR)  and  testing  (XTE)  sets 
have  the  form  of  XXX  (NTR, NVAR)  where  NTR  is  the  number  of  vectors 
and  NVAR  is  the  number  of  net  input  variables.  The  product  of  NTR 
and  NVAR  cannot  exceed  600.  The  program  loops  in  a data  set  and 
uses  a random  number  generator  to  fetch  a data  line  for  the  forma- 
tion of  a vector.  The  program  uses  a flip/flop  and  the  vectors 
are  alternately  thrown  in  the  training  set  and  the  testing  set. 

No  duplicate  vectors  in  a given  data  set  is  permitted. 

The  first  response  on  the  teletype  from  overlay  2 is  "# 
VECTORS:".  The  operator  response  indicates  the  number  of  vectors 
to  be  selected  for  the  training  and/or  testing  set.  The 
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next  teletype  response  is  "MIN  POP:".  The  operator  response 
indicates  the  min  number  of  times  a given  vector  must  be  common 
in  data  set  before  the  vector  is  selected.  The  next  response 
on  the  teletype  is  related  to  maximum  jump  size  (MJS)  and  maxi- 
mum vectors  per  set  (MPPS) . The  program  provides  an  option  to 
select  MJS  and  MPPS  for  each  individual  data  set  or  to  select 
values  for  MJS  and  MPPS  which  are  common  to  all  the  data  sets . 

MPPS  is  the  maximum  number  of  vectors  which  will  be  selected 
from  a given  data  set  and  MJS  is  a number  when  added  to  the  number 
obtained  from  the  random  number  generator  determines  which  data 
line  will  be  fetched  from  a given  data  set.  The  next  teletype 
response  is  "IS  THIS  OK?".  This  is  a program  option  which 
permits  researching  the  data  base. 

A list  of  the  subroutines  and  their  functions  are  given 
below. 

OVERLAY  2 SUBROUTINES 

VCTR  - Sorts  through  designated  data  base  and  forms 
vectors  which  are  a function  of  the  net  input 
parameters 

FETCH  - Previously  described 

RANDU  - Generates  a random  number 

POD  - Calculates  point  estimate  for  a given  vector 

SFBIN  - Divides  surface  finish  into  bins 

TKBIN  - Divides  thickness  into  bins 

CLBIN  - Divides  crack  length  into  bins 

A2CBIN  - Divides  "A/2C”  into  bins,  where  A is  crack  depth 
and  2C  is  crack  length 

MATCHL  - Matches  data  line  with  given  one. 

Overlay  3 

The  function  of  Overlay  3 is  to  set  up  an  independent  test 
problem  with  given  coefficients  for  adaptive  learning.  This  type 
of  approach  was  used  many  times  in  the  initial  development  of 
the  software  to  determine  how  new  ideas  on  fitting  techniques 
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would  improve  either  the  convergence  or  the  fitting  of  the  test 
problem.  The  only  teletype  response  associated  with  this  overlay 
is  an  option  to  use  linear  inputs. 

OVERLAY  3 SUBROUTINES 

LNRIN  - Sets  up  a particular  problem  for  the  adaptive 
learning  to  solve 

RANX  - Sets  up  dimension  and  scaling  factor  for  random 
number  generator 

RANOU  - Previously  described 

LST  - Lists  the  input  vectors. 

Overlay  4 

The  function  of  Overlay  4 is  to  sort  through  both  the 
training  and  testing  sets  and  divide  the  data  into  an  array  of 
the  form  (CL,SF,TK).  For  each  cell  of  this  array,  the  total 
number  of  vectors,  the  total  population  and  the  average  POD  is 
calculated  and  listed  on  the  scope.  An  additional  function  of 
this  overlay  is  to  provide  weighing  factors  in  percent  on 
vectors  having  a population  less  than  a preselected  number. 

The  first  teletype  response  from  Overlay  4 is  program  option 
to  skip  CELPOP.  CELPOP  has  to  be  skipped  when  the  net  input 
variables  does  not  include  surface  finish  and  thickness.  The 
next  response  is  related  to  the  assignment  of  weighting  factors 
below  some  minimum  population.  The  teletype  then  responds  with 
"IS  THIS  OK?".  This  is  a program  option  which  permits  reselect- 
ing the  weighting  factors. 

The  subroutines  and  their  functions  for  this  overlay  are 
listed  below. 

OVERLAY  4 SUBROUTINES 

CELPOP  - Sorts  through  input  data  and  divides  into  cells 
(CL,SF,TH)  and  determines  the  total  number  of 
vectors,  average  POD  and  total  number  of  samples 
for  each  cell. 

DTLST  - Lists  the  CELPOP  results 

CONFCT  - Provides  weighting  factors  for  vectors  having  a 
total  population  equal  to  or  less  than  a given 
minimum  population. 
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Overlay  5 


The  function  of  Overlay  5 is  to  prepare  the  data  for  the 
fitting  program.  First  the  data  is  normalized  by  one  of  two 
methods.  One  method  is  simply  to  divide  each  of  the  net  input 
variables  by  its  corresponding  maximum  value.  The  second  method 
calculates  a mean  and  determines  the  standard  deviation.  The 
input  variable  is  normalized  by  subtracting  its  value  from  the 
mean  and  dividing  by  N standard  deviations  where  N is  a program 
integer  option.  An  additional  function  of  this  overlay  is  to 
perform  a least  square  linear  fit  (LSLF)  of  the  POD  as  a function  of 
each  of  the  net  input  variables.  The  third  function  of  this 
overlay  is  to  initialize  certain  constants  for  the  fitting 
program. 


The  first  teletype  response  from  Overlay  5 is  "NORMALIZE?". 

This  is  a program  option,  which  will  either  work  with  raw  data  or 
normalized  data,  depending  on  the  operator  response.  Assuming 
the  data  is  to  be  normalized  the  next  response  is  "USE  MEAN  & 

STD  DEV?".  This  is  an  option  which  determines  the  method  of 
normalization  to  be  used.  The  next  teletype  responses  are  re- 
lated to  the  LSLF.  Slope,  intercept,  and  standard  deviation 
are  given  for  each  net  input  variable.  The  next  teletype  response 
is  "NORMAZIZE  Y?".  This  is  an  option  to  normalize  y (point 
estimate) . 

The  remaining  teletype  responses  for  this  overlay  are  related 
to  the  initializiation  of  the  training  set  coefficients  as  well 
as  other  constants  related  to  the  fitting  program.  Several  ques- 
tions are  related  to  options  concerning  the  training  set  coef- 
ficients. However  the  best  results  were  obtained  by  doing  an 
iterative  search  on  all  the  coefficients  simultaneously  starting 
all  the  coefficients  initially  at  zero.  The  three  different 
options  permit  (1)  constant  term  coefficients  to  be  zero, 

(2)  nonlinear  coefficients  to  be  zero  and  (3)  set  in  preselected 
coefficients  for  the  linear  terms  on  all  except  the  coefficients 
for  the  first  column  of  nets. 

The  constants  Cl,  C2  and  C3  are  used  to  calculate  a scale 
factor  for  controlling  size  of  the  random  numbers.  AMX  is  also 
used  as  a scale  factor  on  random  numbers  in  the  unguided  search. 
NCOL  is  the  number  of  columns  in  the  network  connectivity.  The 
response  to  "PRINT  INTERVAL"  indicates  frequency  as  function  of 
trials  that  information  related  to  score  of  the  test  set  and 
the  training  set  be  written  out  on  the  teletype.  The  teletype 
response  "USE  RANDOM  NO.  GENERATOR?"  is  an  option  to  initialize 
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the  training  set  coefficients  with  random  numbers.  The  teletype 
response  "WHICH  NET"  permits  selection  of  coefficients  for  any  or 
all  of  the  nets.  Operation  response,  (0,0),  returns  control  to 
the  main  program. 

The  subroutines  and  their  functions  are  listed  below. 

OVERLAY  5 SUBROUTINES 

NRMLZ  - Provides  options  to  provide  different  techniques 

of  normalizing  the  data  and  calls  other  subroutines 
to  prepare  the  data  for  the  adaptive  learning 
fitting  program 

LSLF  - Performs  least  square  linear  fit  for  POD  vs  each 
of  the  net  input  variables 

LNRML  - Normalizes  the  data  by  dividing  each  net  input 
parameter  by  its  corresponding  maximum  value 

NRML  - Calculates  mean  and  standard  deviation  and  nor- 
malize g by  subtracting  the  mean  from  each  net  input 
parameter  value  and  dividing  by  the  standard 
deviation 

PREPAR  - Prepares  input  data 

RANX  - Previously  described. 

Overlay  6 

The  function  of  this  overlay  is  to  perform  an  iterative 
search  over  bounded  space  for  the  test  set  coefficients  for  a 
given  rectangular  connectivity.  The  procedure  employs  an  un- 
guided, guided,  reversal  and  acceleration  with  controlled  step 
size  in  conjunction  with  the  training-testing  paradigm.  Basi- 
cally the  procedure  uses  training  set  vectors  until  an  improve- 
ment in  score  has  been  obtained.  These  same  coefficients  are 
then  applied  to  the  testing  set  vectors  to  check  for  improvement 
in  the  testing  set  score.  The  object  of  the  search  is  to  reduce 
the  test  set  score  to  a minimum.  A flow  diagram  of  this  pro- 
cedure is  given  in  Figure  B-8.  When  the  search  is  terminated,  a 
listing  of  the  final  results  is  displayed  on  the  4010  scope. 
Examples  of  these  diaplays  are  shown  in  Figures  ®-9  and  B-10.  The 
final  results  indicate  the  coefficients  of  each  net;  the  true 
point  estimate,  the  difference  between  the  true  point  estimate 
and  the  predicted  valve,  and  the  weighting  factor  for  each  vector. 
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Figure  3-9  Final  Training  Set  Results 
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Figure  B-10  Final  Testing  Set  Results 


Teletype  responses  related  to  Overlay  6 indicate  seven 
pieces  of  information  related  to  the  iterative  search.  The  in- 
formation indicates  the  type  of  search,  trial  number,  the  best 
to  date  training  score,  the  present  training  score,  the  best  to 
date  test  score,  the  latest  trial  number  where  both  the  training 
and  test  set  scores  improved,  and  the  computed  average  step  size. 

The  subroutines  for  this  overlay  and  their  functions  are 
given  below. 

OVERLAY  6 SUBROUTINES 

FP  - Fitting  program  which  determines  the  coefficients 
of  the  n^ts 

SCORE  - Sum  of  squares  of  the  deviations  between  the 
computed  net  output  and  the  given  POD  value 

NET  - Computes  the  output  of  the  nets  for  a given  set 
of  coefficients 

LSTCOF  - Lists  a set  of  coefficients  for  a given  net 
configuration 


RANX 

AMEND 


VASS 


Previously  described 

Depending  on  an  argument  value,  will  add  two  sets 
of  coefficients,  subtract  two  sets  of  coefficients 
or  multiply  a set  of  coefficients  by  a factor  of 
2 and  add  to  a given  set 

Computes  average  step  size. 


Overlay  7 


The  function  of  this  overlay  is  to  plot  either  the  raw  data 
or  the  derivative  as  a function  of  each  of  the  net  input  vari- 
ables . The  plot  of  the  raw  data  yields  a graphical  representation 
of  the  hypersurface  fit.  Because  of  the  type  of  multinominal 
selected  and  rectangular  connectivity  of  the  network  the  deriva- 
tive remains  a function  of  all  input  variables,  hence  the  deriva- 
tive plot  only  shows  trends . 

The  first  teletype  response  from  Overlay  7 is  "PLOT  RAW 
DATA?".  This  is  an  option  which  will  permit  the  plot  of  raw 
data  or  a plot  of  the  derivatives.  The  next  teletype  response 
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is  "IS  THIS  OK?".  This  is  an  option  which,  depending  on  the 
operator  response,  will  loop  to  "PLOT  RAW  DATA"  option  or  return 
control  to  the  main  program. 

A list  of  Overlay  7 subroutines  and  their  functions  are 
listed  below. 

OVERLAY  7 SUBROUTINES 

DP  - Computes  the  derivative  of  the  POD  as  function  of 
each  of  the  input  variables 

NET  - Previously  described 

NRML  - Previously  described 

RESULT  - Plots  the  derivatives. 

Overlay  8 

The  function  of  Overlay  8 is  to  perform  a parametric  study. 
The  curves  shown  in  the  figures  in  Section  6.1  are  results  of 
this  overlay.  The  plots  trom  the  parametric  study  will  usually 
show  net  output  (predicated  point  estimate)  as  a function  of 
crack  length  while  the  remaining  net  input  variables  are  fixed. 

The  first  teletype  response  from  this  overlay  is  "LOG?". 
This  is  an  option  which,  depending  on  the  operator  response, 
will  show  the  results  as  a semi  log  or  linear  plot.  The  re- 
maining responses  are  related  to  selecting  the  net  input 
variable  for  the  plot  and  fixing  the  remaining  net  input 
variables . 

A list  of  Overlay  8 subroutines  and  their  functions  are 
listed  below. 

OVERLAY  8 SUBROUTINES 

PSP  - Performs  a parametric  study  by  calculating  the  net 
output  from  a fixed  set  of  input  variables  for 
each  vector  ' . 

NET  - Previously  described 

RESULT  - Plots  results  of  the  net  output  as  a function  of 
crack  length. 
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APPENDIX  C 


COMPARISON  OF  STATISTICAL  SCHEMES 
FOR  P.O.D.  CALCULATION 


THE  RANGE  SCHEMES 


The  first  range  scheme  divides  the  total  range  of  crack 
sizes  into  N approximately  equal  subranges.  A binomial 
"lower  95%  confidence  interval"  is  computed  from  the  r^ 

(detected  cracks)  and  m^  (total  cracks  present)  of  the  ith 
interval.  The  smallest  estimated  POD  (Probability  of  De- 
tection) in  this  confidence  interval  is  then  plotted  over 
the  right-hand  endpoint  of  the  crack  size  subrange. 

The  choice  of  N is  arbitrary,  but  by  plotting  the 
lower  estimated  POD  over  the  right-hand  endpoint  of  the 
subrange,  the  smaller  N is  chosen  the  more  conservative 
the  procedure  becomes. 

If  the  lower  estimated  POD  is  plotted  using  this 
range  scheme,  there  is  an  undesirable  consequence.  The 
sample  size  on  which  the  estimate  is  based  can  be  more 
influential  in  determining  the  estimate  than  the  observed 
fraction  of  detections.  For  example, 4 detections  of  4 
cracks  leads  to  a lower  95%  estimate  of  .47,  while  10  de- 
tections of  12  cracks  leads  to  a lower  95%  estimate  of  .56. 

This  scheme  does  not  make  use  of  relevant  information 
from  previous  subranges.  For  example  suppose  that  at  the  .04 
inch  subrange  there  are  90  cracks,  all  of  which  are  detected 
and  at  the  .045  inch  subrange  there  are  6 cracks,  all  of 
which  are  detected. 

The  lower  95%  estimate  for  the  POD  in  the  .04  inch  subrange 
is  .97,  while  for  the  .045  inch  subrange  the  estimate  is  only 
.61.  It  appears  reasonable  to  assume  that  the  larger  the  crack 
size  the  more  likely  the  crack  will  be  detected.  In  the 
example  cited,  surely  the  estimated  POD  at  .045  inch  should  be 
at  least  as  large  as  at  .040  inch.  Some  of  the  other  schemes 
recognize  and  take  advantage  of  additional  relevant  data  bv 
pooling  to  improve  the  precision  of  the  estimators. 

The  second  range  scheme  is  to  divide  the  total  range  of 
crack  sizes  in  subranges  of  unequal  length  but  equal  numbers  of 
cracks.  This  procedure  does  control  variations  in  the  esti- 
mates due  to  sample  size  differences,  however  it  opens  the 
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possibility  of  long  intervals  which  lead  to  conservative 
estimates  as  mentioned  previously.  Like  the  first  range 
scheme,  no  use  is  made  of  relevant  information  from  the 
data  in  smaller  crack  length  subranges. 

In  general  both  these  schemes  are  well  founded 
theoretically,  and  for  cases  in  which  large  sample  sizes 
are  available  for  each  subrange,  their  conservative  features 
present  little  problem.  However,  for  the  present  application 
where  sample  sizes  are  not  always  large,  and  where  relevant 
information  is  available  from  smaller  crack  length  subranges, 
these  procedures  are  overly  conservative. 


"OVERLAPPING  60"  SCHEME 

The  "overlapping  60"  scheme  divides  the  total  range  of 
crack  sizes  into  subranges  which  are  unequal  in  length  and 
are  not  mutually  exclusive.  This  scheme  establishes  the 
largest  crack  length  subrange  by  grouping  the  largest  60  cracks 
together.  The  next  largest  subrange  is  established  by  dropping 
the  largest  30  cracks  from  the  1st  subrange  and  then  acquiring 
the  next  largest  30  cracks  from  the  remainder  of  the  data.  This 
procedure  is  repeated  until  it  is  no  longer  possible  to  acquire 
new  crack  sizes.  After  the  subranges  have  been  established, 
the  binomial  "lower  95°',  confidence  interval"  is  computed,  as 
before,  from  the  r^  (di tected  cracks)  and  m^  (actual  cracks 
present)  in  the  ifcn  subrange.  Again  the  smallest  estimate  of 
the  POD  contained  in  the  confidence  interval  is  plotted  over 
the  right-hand  endpoint  of  the  subrange. 

Since  a subrange  usually  contains  many  different  sizes  of 
crack,  plotting  the  estimated  POD  for  the  subrange  over  the 
largest  crack  size  in  the  subrange  produces  a conservative 
bias  (the  larger  the  subrange  the  greater  the  bias).  This 
scheme,  like  the  second  range  scheme  mentioned,  controls 
variations  in  the  estimates  due  to  sample  size  differences, 
but  allows  the  possibility  of  large  subranges  and  hence 
conservative  estimates  as  mentioned  above.  If  attaining  957. 
confidence  that  the  POD  exceeds  .90  is  deemed  important,  then 
this  procedure  might  be  modified  to  be  an  "overlapping  61"  or 
"overlapping  76"  so  2 and  3 failures  respectively  to  detect 
cracks  would  still  be  compatible  with  the  95/90  criterion, 
with  this  scheme  the  maximum  estimated  POD  is  about  .95  with 
957.  confidence,  ragardless  of  how  well  "large"  cracks  are 


detected.  This  scheme  makes  use  of  information  from  the 
previous  crack  size  subrange  to  compute  the  estimated  POD 
for  the  current  subrange.  This  scheme  does  not  have  the 
option  of  choosing  whether  to  use  data  from  the  previous 
subrange,  it  always  uses  it.  The  consequence  is  that 
for  the  portion  of  the  crack  size  vs.  POD  curve  where  the 
POD  is  increasing  fairly  rapidly,  the  estimated  PODs  are 
likely  to  be  even  mere  conservative  than  the  range  schemes. 
The  portion  of  the  curve  where  the  POD  is  increasing  slowly 
is  likely  to  be  estimated  less  conservatively  than  for  the 
range  schemes. 

In  general  the  procedure  is  conservative,  probably 
more  conservative  than  the  second  range  scheme  in  some 
portions  of  the  POD  curve  and  less  conservative  in  other 
portions  of  the  POD  curve.  This  procedure  is  probably  bette 
in  the  regions  of  most  interest  (threshold  detection  95/90 
size  ranges),  but  overall  is  still  too  conservative. 


OPTIMUM 'PROBABILITY  METHOD  (0PM)  SCHEME 

The  0PM  procedure  divides  the  total  range 
into  N approximately  equal  subranges.  A lower  95% 
confidence  interval  is  computed  from  the  r^  (number  of 
detected  cracks)  and  the  rij  (total  number  of  cracks)  of  the 
largest  subrange.  Next,  the  second  largest  subrange  is 
combined  with  the  largest  subrange  and  a second  lower  con- 
fidence interval  is  computed.  The  process  of  adding  a new 
subrange  and  computing  a new  lower  confidence  interval  is 
repeated  until  all  the  subranges  are  grouped  together.  The 
largest  lower  bound  of  these  N intervals  is  then  plotted 
over  the  right-hand  endpoint  of  the  largest  subrange.  The 
largest  subrange  is  then  eliminated  from  consideration  and 
the  above  procedure  repeated  beginning  with  the  2nd  largest 
subrange.  The  0PM  procedure  is  terminated  when  a 
lower  estimate  of  the  probability  of  detection  has 
been  plotted  for  each  subrange. 

The  0PM  procedure  estimates  a lower  bound  on 
true  probability  of  detection  curve  (true  probability 
of  detection  is  the  conceptual  probability  of  detection 
computed  from  infinite  sampling  information). 
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. However,  the  probability  that  the  lower  estimator  is  less 
than  or  equal  to  the  true  POD  is  .95-8  where  8 > 0 
Therefore  the  confidence  level  is  less  than  957.  and  this 
aspect  of  the  procedure  is  liberal.  As  mentioned  previously, 
plotting  the  estimated  POD  over  the  right-hand  endpoint  of 
the  subrange  is  a conservative  practice. 

A pair  of  simple  examples  will  illustrate  some  of  the  com- 
plexities of  this  problem.  Suppose  that  at  crack  length  .04  in. 
there  are  29  cracks  and  at  crack  length  .035  in.  there  are 
50  cracks.  Further  assume  that  the  POD  at  crack  length  .04  in. 
is  .9  and  also  at  .035  the  POD  is  .9.  In  conceptual  replications 
of  computing  the  lower  "95%  confidence"  interval  for  crack 
length  .040  in.  using  the  0PM  procedure,  about 
92.67%  of  the  intervals  computed  would  contain  .9  (the  true 
POD).  Thus  for  this  case,  the  stated  "95%  confidence"  level  is 
not  achieved.  A closer  examination  of  the  details  of  computing 
the  probability  that  the  estimated  lower  bound  will  be  less  than 
the  true  POD  will  provide  some  useful  insights  about  the  con- 
dition under  which  the  0PM  is  most  liberal.  In  order  to  compute 
the  92.67%  of  intervals  containing  .9  (true  POD),  cited 
previously,  it  is  easiest  to  compute  the  probability  that  the 
interval  will  not  contain  the  POD  of  .9  and  subtract  from  1. 

At  .04  in.  there  are  30  possible  experimental  results:  0,  1, 

29  detected  cracks.  Of  these  results  only  1 leads  to  a lower 
"95%  confidence"  interval  which  does  not  contain  .9,  namely  29 
detected  c racks.  However,  the  0PM  procedure  does  not  necessarily 
use  just  data  gathered  at  .04  in. , it  may  incorporate  data 
gathered  at  .035  in.  The  following  is  a list  of  mutually 
exclusive  experimental  outcomes  which  lead  to  lower  "95% 
confidence"  intervals  which  do  not  contain  .9. 


At 

.04  in. 

At  .035  in. 

1. 

29 

detections 

and 

o, 

or,  1,  or  2,  ...  or  50  detections 

2. 

28 

detections 

and 

48 

or  49  or  50  detections 

3. 

27 

detections 

and 

49  or  50  detections 

4. 

26 

detections 

and 

50  detections 

The 

probabilities 

of  these 

four  mutually  exclusive  experi- 

mental  results  are  computed  (assuming  POD  = .9  at  both  crack 
lengths)  and  added  to  yield  the  probability  of  the  lower  bound 
being  in  error.  Note  that  if  the  POD  at  .035  in.  is  less  than 
.9,  the  probabilities  of  large  numbers  (46,  ...,  50)  of  crack 
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detections  decrease,  which  causes  the  probabilities  of  experi- 
mental results  1 - 4 to  decrease.  The  result  is  that  the 
probability  of  the  lower  bound  being  in  error  decreases.  Since 
it  is  reasonable  to  assume  that  the  POD  increases  with  crack 
size,  keeping  the  POD  constant  for  different  crack  length  ranges 
may  be  viewed  as  a worst  case. 

The  effect  of  having  an  additional  crack  length  range 
for  possible  incorporation  is  illustrated  by  the  next  example. 
Consider  29  cracks  at  .04  in.,  50  cracks  at  .035  in.  and  50 
cracks  at  .03  in.  all  with  a POD  = .9. 

As  in  the  previous  example,  a list  is  compiled  at  mutually 
exclusive  experimental  outcomes  which  lead  to  lower  "957o 
confidence"  intervals  which  do  not  contain  .9. 

# Detections  # Detections  # Detections 


at  . 

04  in. 

at  .035  in. 

at  .030  in. 

1. 

29 

and 

0 or  1 or  ...  49  or  40 

and 

0 or  1 or  ...  49  or  50 

2. 

28 

and 

50 

and 

0 or  1 or  ...  49  or  50 

3. 

28 

and 

49 

and 

0 or  1 or  ...  49  or  50 

4. 

28 

and 

48 

and 

0 or  1 or  ...  49  or  50 

5. 

28 

and 

47 

and 

47  or  48  or  49  or  50 

6. 

28 

and 

46 

and 

48  or  49  or  50 

7. 

28 

and 

45 

and 

49  or  50 

8. 

28 

and 

44 

and 

50 

9. 

27 

and 

50 

and 

0 or  1 or  ...  49  or  50 

10. 

27 

and 

49 

and 

0 or  1 or  ...  49  or  50 

11. 

27 

and 

48 

and 

47  or  48  or  49  or  50 

12. 

27 

and 

47 

and 

48  or  49  or  50 

13. 

27 

and 

46 

and 

49  or  50 

14. 

27 

and 

45 

and 

50 
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# Detections 
at  .04  in. 

# Detections 
at  .035  in. 

# Detections 
at  .030  in. 

15. 

26 

and 

50 

and 

0 or  1 or  ...  49  or  50 

16. 

26 

and 

49 

and 

47 

or  48  or  49  or  50 

17. 

26 

and 

48 

and 

48 

or  49  or  50 

18. 

26 

and 

47 

and 

49 

or  50 

19. 

26 

and 

46 

and 

50 

20. 

25 

and 

50 

and 

47 

or  48  or  49  or  50 

21. 

25 

and 

49 

and 

48 

or  49  or  50 

22. 

25 

and 

48 

and 

49 

or  50 

23. 

25 

and 

47 

and 

50 

24. 

24 

and 

50 

and 

48 

or  49  or  50 

25. 

24 

and 

49 

and 

49 

or  50 

26. 

24 

and 

48 

and 

50 

27. 

23 

and 

50 

and 

49 

or  50 

28. 

23 

and 

49 

and 

50 

29. 

22 

and 

50 

and 

50 

As  before,  the  probabilities  of  these  experimental  outcomes 
are  computed  (assuming  POD  = .9  for  all  three  intervals)  and 
added  to  yield  the  probability,  .0992,  and  the  estimated  lower 
bound  is  in  error.  Thus  there  is  only  about  907o  confidence 
that  the  lower  bound  will  be  lower  than  the  true  POD.  Notice 
that  as  the  number  of  ranges  which  can  potentially  be  incorporated 
increases,  the  potential  error  increases.  The  POD  for  this 
example  was  kept  constant  over  three  ranges  of  crack  length,  so 
this  example  can  be  regarded  as  a "worst  case"  situation. 

There  is  also  a conservative  aspect  to  this  procedure, 
namely  plotting  the  estimated  POD  over  the  right-hand  endpoint 
f f hr>  rrark  ranve.  This  nrnrpHnrp  is  most  ennservatiwe  wViaro 

w 1 

• xii  POD  curve  is  increasing  rapidly  and  least  conservative  when 
POD  curve  is  flat  or  increasing  slowly. 
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Summary:  The  OPM  scheme  is  a powerful  procedure  (makes  the 
most  of  available  data)  which  has  a small  liberal  bias 
(the  confidence  is  smaller  than  the  stated  95%) . The  size 
of  the  bias  depends  on: 

L True  shape  of  POD  curve 
2.  Sample  sizes  in  different  ranges 
3o  Total  number  of  ranges 

4„  Where  in  the  interval  the  estimated  POD  is  plotted 

Subjective  Evaluation:  Overall  probably  the  best  procedure 
because  it  handles  the  unequal  sample  size  problem  and  makes 
full  use  of  available  data.  The  price  that  is  paid  for  using 
this  procedure  is  the  liberal  bias  in  crack  size  regions  where 
the  POD  curve  is  flat  or  increasing  slowly. 
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EXAMPLES  OF  COMPUTER  GENERATED 
HISTOGRAMS  OBTAINED  BY  THE  POINT  ESTIMATE  METHOD 
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EXAMPLES  OF  COMPUTER  GENERATED  HISTOGRAMS  OBTAINED 
BY  THE  POINT  ESTIMATE  METHOD 


The  effect  of  specimen  thickness  on  the  POD  was  shown  earlier 
in  Figure  3 in  the  test.  Figures  D-I  and  D-2  present  a comparison 
of  the  PODs  for  specimens  with  thicknesses  of  0.214  and  0.060  in. 
with  three  ranges  of  surface  finishes  for  eddy  current  and  ultrasonic 
inspection  techniques  respectively.  Figure  D-3  presents  similar 
comparison  for  specimens  with  four  ranges  of  surface  finishes  for 
the  penetrant  technique.  Comparisons  for  specimens  with  two  ranges 
of  surface  finishes  are  shown  in  Figure  D-4  for  X-ray  inspection. 

The  comparisons  in  Figures  D-l  to  D-4  indicated  that  specimen 
surface  finish  has  negligible  effect  on  the  POD  for  ultrasonic  and 
X-ray  methods.  For  eddy  current  and  penetrant  inspections  the 
effect  is  noticeable  only  in  the  smallest  crack  length  region  of 
0 - 0.125  in.  In  both  cases  only  the  smoothest  range  of  surface 
finish  of  0-32  RMS  appeared  to  have  different  effects  on  the  POD 
compared  to  the  other  surface  finish  ranges.  For  eddy  current 
inspection  the  POD  for  the  smoothest  surface  finish  is  highest 
in  the  0-0.125  in.  crack  length.  For  penetrant  inspection  the 
POD  for  the  smoothest  surface  finish  range  is  lowest  in  the 
smallest  crack  length  range.  Comparing  specimen  thicknesses,  the 
only  significant  difference  in  POD  for  specimens  with  two  different 
thicknesses  was  observed  for  the  X-ray  technique. 

Figures  D-5,  D-6,  D-7  and  D-8  present  comparisons  of  POD  for 
specimens  with  etched  and  unetched  surfaces  for  eddy  current, 
ultrasonic,  penetrant  and  X-ray  technique  respectively.  For 
the  first  three  NDE  techniques,  results  of  flat  plate  specimens, 
integrally  stiffened  panels  and  a composite  of  these  two  specimen 
types  from  three  different  companies  are  presented  separately. 

For  the  X-ray  inspection,  only  the  flat  plate  specimens  have 
sufficient  data  for  comparison  purpose.  Results  presented  in  these 
figures  indicated  that  etching  of  the  specimen  surfaces  increased 
the  POD  for  all  NDT  techniques  except  ultrasonics.  The  effect  of 
etching  in  the  penetrant  technique  appeared  to  be  more  pronounced 
for  flat  plate  specimens  as  compared  to  integrally  stiffened  panels 
and  composites.  For  eddy  current  technique,  the  effect  of  etching 
appeared  to  be  more  pronounced  for  the  integrally  stiffened  panels 
and  composite  results  as  compared  to  flat  specimens.  No  noticeable 
effect  could  be  discerned  from  the  etching  for  the  ultrasonic 
technique . 

A comparison  of  f-h«»  Pnn  for  different  inspectors  within  the 
same  company  is  presented  in  Figure  'D-9  for  penetrant,  eddy 
current,  and  X-ray  techniques.  The  inspection  efficiency  of  the 
same  class  of  inspectors  within  the  same  company  appears  to  be 
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sp;  I-32  RMS  SF : 33-64  RMS  SF:  129-250 


Figure  D-l  Comparison  of  POD  of  Flat  Plate  Specimen  Thickness  (Eddy  Current) 
(1st  and  2nd  Columns  Represent  Specimens  with  Thicknesses  of  0.214 
and  0.060  In.  Respectively.  DSN  = 15-18,  31) 


and  0.060  In.  Respectively.  DSN  = 12,13,14,29,30) 


st  and  2nd  Columns  Represent  Specimens  with  Thicknesses  of 
214  and  0.060  In.  Respectively.  DSN  = 24-28,  60,  61) 


2nd  Columns  Represent  Specimens  with  Etched  and  Unetched 
Surface  Treatment  Respectively) 


(1st,  2nd  and  3rd  Columns  Represent  Different  Inspectors 
within  One  Company) 


quite  uniform.  For  the  penetrant  technique,  the  inspector  who 
generated  data  set  No.  28  is  judged  to  be  below  par.  The 
inspector  who  generated  data  set  No.  18  for  the  eddy  current 
technique  had  higher  POD  compared  to  the  other  two  inspectors 
in  the  same  company,  at  least  in  the  longer  crack  length  ranges. 

The  inspector  who  generated  data  set  No.  20  in  the  X-ray  tech- 
nique is  judged  to  be  below  par.  For  the  ultrasonic  technique, 
different  inspectors  did  not  produce  any  noticeable  difference 
in  POD.  Figure  D-10  presents  a comparison  of  the  POD  on  flat 
plate  specimens  containing  fatigue  cracks  obtained  by  three 
companies  coded  A,  B and  C using  penetrant,  ultrasonic,  eddy 
current,  and  X-ray  techniques.  The  POD  for  the  first  three 
techniques  appeared  to  be  quite  similar.  The  largest  variation 
in  POD  existed  in  the  inspection  results  for  the  X-ray  technique. 

The  effects  of  specimen  geometry  on  POD  for  four  NDE  tech- 
niques are  presented  in  Figures  D-ll,  D-12  and  D-13.  PODs 
obtained  on  flat  plate  specimens  and-  integrally  stiffened  panels 
containing  fatigue  cracks  obtained  by  using  penetrant,  eddy 
current  and  ultrasonic  inspections  are  shown  in  Figure  D-ll.  It 
appears  that  the  POD  for  the  simpler  geometry  were  higher  than 
those  for  the  more  complex  geometry  for  all  three  NDT  techniques. 
However,  the  difference  was  significant  only  for  the  smallest 
crack  length  range  in  the  case  of  ultrasonic  inspection.  Figure 
D-12  presents  a comparison  of  the  PODs  obtained  on  flat  plates, 
integrally  stiffened  panel  (ISP)  and  ISP  with  riveted  plate  for 
the  same  three  NDT  techniques  of  penetrant,  eddy  current,  and 
ultrasonics.  For  all  three  NDT  techniques,  PODs  obtained  on  flat 
plates  were  higher  than  corresponding  values  for  the  other  two 
types  of  specimens  with  more  complex  geometries.  However,  little 
difference  could  be  discerned  for  the  integrally  stiffened  panels 
with  and  without  a riveted  plate.  Figure  D-13  shows  a comparison 
of  the  POD  obtained  on  specimens  with  several  part  geometries  by 
using  ultrasonic,  eddy  current,  and  magnetic  particle  techniques. 

In  general,  no  significant  difference  was  observed  for  the  specimen 
geometries  represented  except  the  tandem  T versus  hollow  straight 
cylinder.  However,  the  reason  for  the  low  POD  for  the  latter  was 
that  the  inspectors  were  not  aware  that  cracks  were  present  on  the 
inner  surfaces  of  the  hollow  cylinders. 

A comparison  of  POD  for  weld  specimens  with  as-welded  and 
scarfed  joints  using  penetrant,  ultrasonic,  and  eddy  current 
techniques  is  shown  in  Figure  D-14.  The  PODs  for  the  two  types 
of  specimen  histories  were  essentially  equivalent  for  the  ultra- 
sonic and  eddy  current  techniques.  For  the  penetrant  technique. 
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Figure  D-10  Comparison  of  POD  for  Different  Companies  Using  the  Same 
Flat  Plate  Specimens  (1st,  2nd,  and  3rd  Columns  Represent 
Companies  A,  B,  and  C Respectively) 


Represent  Specimen  Geometries  of  Flat  Plate,  Integra 
Stiffened  Panel  (ISP),  and  ISP  with  Riveted  Plate 
Respectively) 


lower  PODs  were  evident  for  specimens  with  scarfed  joints.  This 
appeared  to  be  contradictory  to  the  expected  trend.  The  reason 
for  the  anomaly  could  be  attributed  to  a smearing  of  the  scarfed 
surfaces  of  the  aluminum  weld  specimen.  The  flaw  openings  to  the 
specimen  surface  were  closed  by  the  metal  chips  preventing  the 
penetration  of  the  penetrants.  Figure  D-15  presents  a comparison 
of  POD  for  weld  specimens  with  lack  of  penetration  (LOP)  defects 
inspected  before  and  after  a proof  loading  of  90  percent  of  yield 
stress.  Except  for  the  eddy  current  technique,  the  PODs  obtained 
for  the  specimens  after  proof  loading  were  much  higher  compared  to 
those  obtained  before  proof  loading. 

The  point  estimate  comparison  of  the  PODs  presented  in 
Figures  D-7  to  D-15  provided  a good  indication  of  the  order  of 
importance  of  the  influencing  parameters  on  the  NDE  sensitivity. 
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LINEAR  REGRESSION  TRANSLATION  MODELS 


LINEAR  REGRESSION  TRANSLATION  MODEL 


The  following  linear  regression  translation  models  were 
obtained  by  ucing  procedures  outlined  in  Section  5.1.2  in  this 
report.  The  tables  were  categorized  by  the  complex  geometries 
to  which  results  obtained  from  flat  plate  specimens  were 
tabulated.  Two  sets  of  data  were  associated  with  each  category 
of  complex  specimen  geometry.  The  first  set  labelled  "Crack 
Length"  was  obtained  without  entering  the  crack  depth  as  a 
parameter.  The  second  set  labelled  "Crack  Depth"  was  obtained 
after  the  crack  depth  to  crack  length  (a/2c)  ratio  was  entered 
as  a parameter.  The  first  row  of  numbers  in  the  translation 
model  represented  the  coefficients  accounting  for  the  contri- 
bution to  the  POD  from  the  geometry  effect.  The  second  block 
of  numbers  tabulated  the  contributions  from  nine  crack  length 
intervals.  Contribution  from  the  specimen  history,  surface 
finish,  and  the  a/2c  of  the  specimens  are  shown  in  the  third, 
fourth  and  fifth  blocks  of  coefficients  respectively  in  the 
tables . 
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Fort  Worth  Division  TABLE  El  LINEAR  REGRESSION  MODEL  - PENETRANT 

ABULATION  SHEET 


Fort  Worth  Division  TABLE  E 4 LINEAR  REGRESSION  MODEL  - X-RAY 

TABULATION  SHEET 
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